Oct 31 13:02:36.809910 ip-10-0-128-41 systemd[1]: Starting Kubernetes Kubelet... Oct 31 13:02:37.225894 ip-10-0-128-41 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 13:02:37.225894 ip-10-0-128-41 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 13:02:37.225894 ip-10-0-128-41 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 13:02:37.225894 ip-10-0-128-41 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 13:02:37.230513 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230424 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:02:37.230513 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230509 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:02:37.230513 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230514 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:02:37.230513 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230519 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230524 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230528 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230533 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230537 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230541 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230545 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230549 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230554 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230558 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230562 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230566 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230571 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230576 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230579 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230583 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230587 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230606 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230611 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:02:37.230801 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230615 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230620 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230624 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230628 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230631 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230635 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230639 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230643 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230647 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230652 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230656 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230660 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230664 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230668 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230672 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230678 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230682 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230687 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230691 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230695 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:02:37.231563 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230699 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230703 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230707 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230712 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230718 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230722 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230726 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230730 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230735 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230740 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230746 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230752 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230757 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230762 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230766 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230771 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230775 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230780 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230784 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:02:37.232455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230789 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230793 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230797 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230802 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230806 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230809 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230814 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230820 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230825 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230829 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230834 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230839 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230843 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230848 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230852 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230856 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230860 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230865 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230871 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230875 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:02:37.233268 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230880 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230885 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230889 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230893 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230898 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230905 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230912 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230917 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230923 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230927 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230932 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230936 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230940 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230945 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230949 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230953 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230957 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230962 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230966 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:02:37.234131 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230971 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230978 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230982 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230987 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230991 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.230997 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232293 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232303 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232308 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232313 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232318 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232322 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232327 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232331 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232336 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232340 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232345 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232350 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232354 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232358 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:02:37.234949 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232362 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232366 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232370 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232374 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232379 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232384 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232388 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232392 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232397 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232401 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232405 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232409 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232413 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232418 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232423 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232427 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232431 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232435 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232440 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:02:37.235946 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232444 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232448 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232452 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232456 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232460 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232464 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232469 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232473 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232477 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232481 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232486 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232490 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232495 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232499 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232503 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232507 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232512 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232516 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232520 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232524 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:02:37.236751 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232528 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232533 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232537 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232542 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232546 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232550 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232555 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232559 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232564 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232569 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232574 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232578 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232583 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232587 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232611 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232616 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232620 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232624 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232628 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:02:37.237442 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232633 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232638 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232642 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232647 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232652 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232656 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232660 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232664 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232669 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232673 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232677 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232681 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232687 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232691 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232696 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232701 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232705 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232709 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232713 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232717 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:02:37.237956 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232721 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232726 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232730 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232737 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232743 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232747 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232752 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232756 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232760 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232767 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232774 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232780 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232784 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.232792 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233636 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233646 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233651 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233656 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233660 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:02:37.238523 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233664 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233669 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233673 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233678 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233683 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233688 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233692 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233696 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233701 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233705 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233709 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233714 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233718 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233722 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233727 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233731 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233735 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233739 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233744 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233748 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:02:37.239020 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233752 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233756 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233761 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233766 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233770 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233774 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233780 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233784 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233788 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233793 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233797 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233801 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233806 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233810 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233814 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233818 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233822 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233830 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233836 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:02:37.239517 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233840 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233845 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233850 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233855 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233859 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233863 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233868 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233872 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233876 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233880 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233884 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233891 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233895 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233900 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233904 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233908 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233913 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233917 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233921 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233925 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:02:37.240018 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233930 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233935 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233939 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233943 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233947 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233952 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233956 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233960 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233964 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233968 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233973 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233980 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233986 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233992 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.233997 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234001 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234005 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234010 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234014 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:02:37.240488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234018 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234022 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234027 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234031 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234038 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234042 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234046 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234051 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234055 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234060 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234064 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234068 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234072 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234076 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234082 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234086 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234090 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234095 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234099 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:02:37.240981 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234103 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234108 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234112 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.234116 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234249 2576 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234260 2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234270 2576 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234277 2576 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234285 2576 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234290 2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234297 2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234304 2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234309 2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234314 2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234320 2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234326 2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234331 2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234338 2576 flags.go:64] FLAG: --cgroup-root="" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234343 2576 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234348 2576 flags.go:64] FLAG: --client-ca-file="" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234352 2576 flags.go:64] FLAG: --cloud-provider="external" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234357 2576 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234365 2576 flags.go:64] FLAG: --cluster-domain="" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234369 2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 13:02:37.241514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234374 2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234379 2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234385 2576 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234391 2576 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234402 2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234408 2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234413 2576 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234418 2576 flags.go:64] FLAG: --contention-profiling="false" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234423 2576 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234428 2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234434 2576 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234438 2576 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234454 2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234458 2576 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234463 2576 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234468 2576 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234473 2576 flags.go:64] FLAG: --enable-server="true" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234478 2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234485 2576 flags.go:64] FLAG: --event-burst="100" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234490 2576 flags.go:64] FLAG: --event-qps="50" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234495 2576 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234499 2576 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234504 2576 flags.go:64] FLAG: --eviction-hard="" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234511 2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 13:02:37.242135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234516 2576 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234523 2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234528 2576 flags.go:64] FLAG: --eviction-soft="" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234533 2576 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234538 2576 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234543 2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234548 2576 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234552 2576 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234557 2576 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234562 2576 flags.go:64] FLAG: --feature-gates="" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234569 2576 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234574 2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234579 2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234585 2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234606 2576 flags.go:64] FLAG: --healthz-port="10248" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234612 2576 flags.go:64] FLAG: --help="false" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234617 2576 flags.go:64] FLAG: --hostname-override="ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234622 2576 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234627 2576 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234632 2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234640 2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234646 2576 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234650 2576 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 13:02:37.242780 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234655 2576 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234660 2576 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234665 2576 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234669 2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234675 2576 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234679 2576 flags.go:64] FLAG: --kube-reserved="" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234684 2576 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234689 2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234694 2576 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234699 2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234705 2576 flags.go:64] FLAG: --lock-file="" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234710 2576 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234715 2576 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234727 2576 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234739 2576 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234743 2576 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234749 2576 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234754 2576 flags.go:64] FLAG: --logging-format="text" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234758 2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234764 2576 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234769 2576 flags.go:64] FLAG: --manifest-url="" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234774 2576 flags.go:64] FLAG: --manifest-url-header="" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234781 2576 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234786 2576 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234793 2576 flags.go:64] FLAG: --max-pods="110" Oct 31 13:02:37.243532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234797 2576 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234802 2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234807 2576 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234812 2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234818 2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234823 2576 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234828 2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234841 2576 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234846 2576 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234851 2576 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234855 2576 flags.go:64] FLAG: --pod-cidr="" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234860 2576 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234865 2576 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234870 2576 flags.go:64] FLAG: --pods-per-core="0" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234875 2576 flags.go:64] FLAG: --port="10250" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234879 2576 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234884 2576 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-042410bbffce94d1b" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234891 2576 flags.go:64] FLAG: --qos-reserved="" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234896 2576 flags.go:64] FLAG: --read-only-port="10255" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234901 2576 flags.go:64] FLAG: --register-node="true" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234906 2576 flags.go:64] FLAG: --register-with-taints="" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234912 2576 flags.go:64] FLAG: --registry-burst="10" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234917 2576 flags.go:64] FLAG: --registry-qps="5" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234921 2576 flags.go:64] FLAG: --reserved-cpus="" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234926 2576 flags.go:64] FLAG: --reserved-memory="" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234932 2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 13:02:37.244398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234937 2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234942 2576 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234947 2576 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234952 2576 flags.go:64] FLAG: --runonce="false" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234956 2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234961 2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234966 2576 flags.go:64] FLAG: --seccomp-default="false" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234971 2576 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234976 2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234981 2576 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234988 2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234993 2576 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.234998 2576 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235003 2576 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235007 2576 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235012 2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235017 2576 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235021 2576 flags.go:64] FLAG: --system-cgroups="" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235026 2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235035 2576 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235039 2576 flags.go:64] FLAG: --tls-cert-file="" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235044 2576 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235051 2576 flags.go:64] FLAG: --tls-min-version="" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235057 2576 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235062 2576 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 13:02:37.245037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235067 2576 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235072 2576 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235077 2576 flags.go:64] FLAG: --v="2" Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235083 2576 flags.go:64] FLAG: --version="false" Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235090 2576 flags.go:64] FLAG: --vmodule="" Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235096 2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235102 2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235326 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235333 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235339 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235344 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235348 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235353 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235359 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235363 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235367 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235371 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235377 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235383 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235387 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235391 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:02:37.245644 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235396 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235400 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235404 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235408 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235413 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235417 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235421 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235426 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235431 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235437 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235441 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235445 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235449 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235453 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235458 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235462 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235466 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235470 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235475 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:02:37.246149 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235479 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235483 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235487 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235492 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235496 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235500 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235505 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235509 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235513 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235519 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235522 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235527 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235532 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235536 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235539 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235543 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235548 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235552 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235556 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235560 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:02:37.246674 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235564 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235568 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235573 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235577 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235581 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235587 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235611 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235617 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235622 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235627 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235631 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235635 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235639 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235643 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235647 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235652 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235656 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235659 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235664 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235670 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:02:37.247157 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235674 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235680 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235684 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235688 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235693 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235697 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235701 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235706 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235710 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235715 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235719 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235723 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235727 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235735 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235742 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235747 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235752 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235756 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235760 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235765 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:02:37.247671 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235769 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235773 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235778 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235782 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235786 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235790 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235795 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235799 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235803 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235807 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235812 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235816 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235822 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.235913 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235971 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235978 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:02:37.248135 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235984 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235989 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.235995 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236000 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236005 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236009 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236014 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236020 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236027 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236032 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236037 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236041 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236046 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236050 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236054 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236058 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236062 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236066 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236071 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236075 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:02:37.248521 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236079 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236083 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236088 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236092 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236097 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236101 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236105 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236109 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236114 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236121 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236126 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236130 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236134 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236138 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236143 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236149 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236155 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236160 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236165 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236169 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:02:37.249011 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236174 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236178 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236183 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236187 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236191 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236196 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236200 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236205 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236209 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236213 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236218 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236222 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236226 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236231 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236235 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236239 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236243 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236247 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236252 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:02:37.249488 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236256 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236260 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236267 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236272 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236277 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236281 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236285 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236290 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236294 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236299 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236303 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236307 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236312 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236316 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236320 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236324 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236329 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236333 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236338 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:02:37.249961 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236342 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236346 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236350 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236355 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236359 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236363 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236367 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236371 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236375 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236379 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236383 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236388 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236392 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236396 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236400 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236406 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236410 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236415 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236419 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:02:37.250409 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236424 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236428 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236432 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236439 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236443 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236447 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.236451 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.243306 2576 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.243324 2576 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.243347 2576 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.243362 2576 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243694 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243705 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243711 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243727 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243733 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243739 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243744 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243750 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:02:37.250876 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243756 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243762 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243768 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243773 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243777 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243783 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243787 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243797 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243802 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243808 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243813 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243818 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243825 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243831 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243865 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243871 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243876 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243881 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243885 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:02:37.251331 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243896 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243901 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243905 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243909 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243914 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243919 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243923 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243928 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243933 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243938 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243943 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243947 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243951 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243961 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243965 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243970 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243974 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243979 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243983 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:02:37.251841 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243988 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243992 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.243997 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244001 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244005 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244010 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244015 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244026 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244030 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244058 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244121 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244274 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244280 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244284 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244288 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244291 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244297 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244301 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244304 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244306 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:02:37.252288 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244309 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244312 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244315 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244318 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244321 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244324 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244326 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244329 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244332 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244334 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244337 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244339 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244342 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244345 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244348 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244351 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244353 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244356 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244358 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244361 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:02:37.252807 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244364 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244367 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244369 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244374 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244377 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244379 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244382 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244385 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244387 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244390 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244393 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244395 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244398 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244400 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244403 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244406 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244409 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244412 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244414 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:02:37.253291 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244418 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.244454 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244652 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244658 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244662 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244665 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244668 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244670 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244673 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244675 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244678 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244681 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244684 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244686 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244689 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244692 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:02:37.253759 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244696 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244699 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244701 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244704 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244708 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244710 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244713 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244716 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244719 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244721 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244724 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244726 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244730 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244734 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244737 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244740 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244743 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244745 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244748 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:02:37.254133 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244750 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244753 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244756 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244759 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244761 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244764 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244766 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244769 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244771 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244774 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244776 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244779 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244783 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244787 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244790 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244793 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244796 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244798 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244801 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:02:37.254569 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244803 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244806 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244808 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244811 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244813 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244816 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244818 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244821 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244824 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244826 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244829 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244832 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244834 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244838 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244841 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244843 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244846 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244848 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244851 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244854 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:02:37.255108 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244857 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244860 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244862 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244865 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244868 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244870 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244873 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244876 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244878 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244881 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244884 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244886 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244889 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244891 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244894 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244896 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244898 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244901 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244904 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244906 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:02:37.255575 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244909 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244911 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244914 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244916 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244919 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244921 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244924 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244926 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244929 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244931 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244933 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244939 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244942 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.244945 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.244970 2576 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245001 2576 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:02:37.256090 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245005 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245008 2576 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245011 2576 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245014 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245017 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245019 2576 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245022 2576 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245025 2576 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245028 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245030 2576 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245033 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245035 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245038 2576 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245041 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245045 2576 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245048 2576 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245052 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245055 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245066 2576 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:02:37.256455 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245070 2576 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245072 2576 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245075 2576 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245077 2576 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245080 2576 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245083 2576 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245086 2576 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245088 2576 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245091 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245095 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245099 2576 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245102 2576 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245104 2576 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245107 2576 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245110 2576 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245113 2576 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245115 2576 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245118 2576 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245120 2576 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245123 2576 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:02:37.256933 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245126 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245128 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245131 2576 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245134 2576 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245136 2576 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245139 2576 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245141 2576 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245144 2576 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245146 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245149 2576 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245151 2576 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245155 2576 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245157 2576 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245160 2576 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245162 2576 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245165 2576 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245167 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245170 2576 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245172 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245175 2576 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:02:37.257422 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245177 2576 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245182 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245185 2576 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245188 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245191 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245193 2576 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245196 2576 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245200 2576 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245202 2576 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245205 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245208 2576 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245210 2576 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245213 2576 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245216 2576 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245218 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245221 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245223 2576 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245226 2576 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245229 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:02:37.257897 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245231 2576 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245234 2576 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245237 2576 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245239 2576 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245242 2576 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245244 2576 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245247 2576 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245249 2576 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245252 2576 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245254 2576 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245257 2576 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245259 2576 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245262 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245264 2576 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245269 2576 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245271 2576 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245274 2576 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245277 2576 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245280 2576 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245283 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:02:37.258337 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245286 2576 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:02:37.258825 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245289 2576 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:02:37.258825 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245291 2576 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:02:37.258825 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245296 2576 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:02:37.258825 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245300 2576 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:02:37.258825 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245303 2576 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:02:37.258825 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:37.245306 2576 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:02:37.258825 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.245367 2576 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 13:02:37.258825 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.247299 2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 31 13:02:37.258825 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.248215 2576 server.go:1014] "Starting client certificate rotation" Oct 31 13:02:37.258825 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.248307 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 13:02:37.258825 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.248337 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 13:02:37.258825 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.250110 2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 13:02:37.258825 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.251281 2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 13:02:37.264325 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.264309 2576 log.go:25] "Validated CRI v1 runtime API" Oct 31 13:02:37.269927 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.269911 2576 log.go:25] "Validated CRI v1 image API" Oct 31 13:02:37.270751 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.270728 2576 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 13:02:37.274059 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.274042 2576 fs.go:135] Filesystem UUIDs: map[23873c31-c9aa-48a6-9630-edd3a87c7243:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a13bc9ed-8022-437b-a32c-b640aa1f4297:/dev/nvme0n1p3] Oct 31 13:02:37.274109 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.274060 2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 31 13:02:37.280658 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.280259 2576 manager.go:217] Machine: {Timestamp:2026-10-31 13:02:37.27834647 +0000 UTC m=+0.341143847 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3093813 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24479514f8e05737fad8540417e159 SystemUUID:ec244795-14f8-e057-37fa-d8540417e159 BootID:7b835058-1da9-4c2c-9948-d949b1b39189 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fc:18:43:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fc:18:43:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:a6:60:5d:dd:63:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 13:02:37.280658 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.280655 2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 13:02:37.280845 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.280821 2576 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 13:02:37.283639 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.283611 2576 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 13:02:37.283766 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.283641 2576 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-41.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 13:02:37.283820 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.283769 2576 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 13:02:37.283820 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.283777 2576 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 13:02:37.283820 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.283818 2576 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 13:02:37.283897 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.283836 2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 13:02:37.284461 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.284451 2576 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 13:02:37.284973 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.284956 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 13:02:37.287829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.287816 2576 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 13:02:37.287950 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.287940 2576 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 13:02:37.287995 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.287972 2576 kubelet.go:493] "Attempting to sync node with API server" Oct 31 13:02:37.287995 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.287986 2576 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 13:02:37.287995 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.287996 2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 13:02:37.288104 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.288005 2576 kubelet.go:405] "Adding apiserver pod source" Oct 31 13:02:37.288104 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.288014 2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 13:02:37.288421 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.288402 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 13:02:37.290474 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.290454 2576 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 13:02:37.294662 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.294648 2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 13:02:37.295444 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.295428 2576 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 13:02:37.295515 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.295459 2576 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 13:02:37.295667 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.295653 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 13:02:37.295701 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.295684 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 13:02:37.295701 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.295697 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 13:02:37.295757 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.295707 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 13:02:37.295757 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.295716 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 13:02:37.295757 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.295731 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 13:02:37.296224 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.296214 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 13:02:37.296224 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.296224 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 13:02:37.296287 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.296239 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 13:02:37.296287 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.296245 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 13:02:37.296287 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.296254 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 13:02:37.296287 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.296262 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 13:02:37.296771 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.296761 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 13:02:37.296803 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.296775 2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 13:02:37.300635 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.300613 2576 server.go:1263] "Started kubelet" Oct 31 13:02:37.300728 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.300660 2576 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 13:02:37.300800 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.300739 2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 13:02:37.300845 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.300835 2576 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 13:02:37.301562 ip-10-0-128-41 systemd[1]: Started Kubernetes Kubelet. Oct 31 13:02:37.303180 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.302052 2576 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 13:02:37.303180 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.302945 2576 server.go:317] "Adding debug handlers to kubelet server" Oct 31 13:02:37.306466 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.306440 2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-41.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 31 13:02:37.307314 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.307282 2576 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 31 13:02:37.307688 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.306435 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-41.us-east-2.compute.internal.18e39edf03b31290 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-41.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-41.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 13:02:37.300552336 +0000 UTC m=+0.363349713,LastTimestamp:2026-10-31 13:02:37.300552336 +0000 UTC m=+0.363349713,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-41.us-east-2.compute.internal,}" Oct 31 13:02:37.310973 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.310942 2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 13:02:37.311937 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.311923 2576 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 13:02:37.312019 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.311994 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:37.312406 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.312390 2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 13:02:37.312483 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.312392 2576 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 13:02:37.312483 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.312427 2576 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 13:02:37.312619 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.312511 2576 reconstruct.go:97] "Volume reconstruction finished" Oct 31 13:02:37.312619 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.312519 2576 reconciler.go:29] "Reconciler: start to sync state" Oct 31 13:02:37.312761 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.312745 2576 factory.go:55] Registering systemd factory Oct 31 13:02:37.312798 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.312766 2576 factory.go:223] Registration of the systemd container factory successfully Oct 31 13:02:37.313029 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.313007 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 31 13:02:37.313081 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.313025 2576 factory.go:153] Registering CRI-O factory Oct 31 13:02:37.313081 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.313045 2576 factory.go:223] Registration of the crio container factory successfully Oct 31 13:02:37.313134 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.313083 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-41.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 31 13:02:37.313134 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.313108 2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 13:02:37.313134 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.313129 2576 factory.go:103] Registering Raw factory Oct 31 13:02:37.313211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.313148 2576 manager.go:1199] Started watching for new ooms in manager Oct 31 13:02:37.314070 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.314047 2576 manager.go:319] Starting recovery of all containers Oct 31 13:02:37.322576 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.322554 2576 manager.go:324] Recovery completed Oct 31 13:02:37.323270 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.323240 2576 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-41.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 31 13:02:37.329550 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.329495 2576 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 31 13:02:37.331089 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.331075 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 13:02:37.333138 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.333122 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 13:02:37.333200 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.333150 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 13:02:37.333200 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.333164 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 13:02:37.333810 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.333730 2576 cpu_manager.go:226] "Starting" policy="none" Oct 31 13:02:37.333810 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.333743 2576 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 13:02:37.333810 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.333742 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 13:02:37.333810 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.333764 2576 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 13:02:37.334470 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.334450 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 13:02:37.334531 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.334481 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 13:02:37.334531 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.334491 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 13:02:37.339220 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.339153 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-41.us-east-2.compute.internal.18e39edf05a446b2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-41.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-41.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-41.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 13:02:37.333137074 +0000 UTC m=+0.395934450,LastTimestamp:2026-10-31 13:02:37.333137074 +0000 UTC m=+0.395934450,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-41.us-east-2.compute.internal,}" Oct 31 13:02:37.341371 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.341354 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l2vfp" Oct 31 13:02:37.350226 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.350147 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-41.us-east-2.compute.internal.18e39edf05a48d41 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-41.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-41.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-41.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 13:02:37.333155137 +0000 UTC m=+0.395952513,LastTimestamp:2026-10-31 13:02:37.333155137 +0000 UTC m=+0.395952513,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-41.us-east-2.compute.internal,}" Oct 31 13:02:37.354040 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.354020 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-l2vfp" Oct 31 13:02:37.364024 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.363956 2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-41.us-east-2.compute.internal.18e39edf05a4c658 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-41.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-41.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-41.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 13:02:37.333169752 +0000 UTC m=+0.395967131,LastTimestamp:2026-10-31 13:02:37.333169752 +0000 UTC m=+0.395967131,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-41.us-east-2.compute.internal,}" Oct 31 13:02:37.379301 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.379234 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-41.us-east-2.compute.internal.18e39edf05a446b2\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-41.us-east-2.compute.internal.18e39edf05a446b2 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-41.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-128-41.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-128-41.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 13:02:37.333137074 +0000 UTC m=+0.395934450,LastTimestamp:2026-10-31 13:02:37.334468713 +0000 UTC m=+0.397266095,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-41.us-east-2.compute.internal,}" Oct 31 13:02:37.392919 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.392836 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-41.us-east-2.compute.internal.18e39edf05a48d41\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-41.us-east-2.compute.internal.18e39edf05a48d41 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-41.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-128-41.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-128-41.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 13:02:37.333155137 +0000 UTC m=+0.395952513,LastTimestamp:2026-10-31 13:02:37.334486238 +0000 UTC m=+0.397283617,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-41.us-east-2.compute.internal,}" Oct 31 13:02:37.475206 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.409930 2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-128-41.us-east-2.compute.internal.18e39edf05a4c658\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-41.us-east-2.compute.internal.18e39edf05a4c658 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-41.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-128-41.us-east-2.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-128-41.us-east-2.compute.internal,},FirstTimestamp:2026-10-31 13:02:37.333169752 +0000 UTC m=+0.395967131,LastTimestamp:2026-10-31 13:02:37.334495178 +0000 UTC m=+0.397292556,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-41.us-east-2.compute.internal,}" Oct 31 13:02:37.475206 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.413033 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:37.475206 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.447263 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 13:02:37.475206 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.448443 2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 13:02:37.475206 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.448470 2576 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 13:02:37.475206 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.448516 2576 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 13:02:37.475206 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.448556 2576 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 13:02:37.475206 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.473830 2576 policy_none.go:50] "Start" Oct 31 13:02:37.475206 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.473851 2576 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 13:02:37.475206 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.473866 2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 13:02:37.477871 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.475836 2576 policy_none.go:44] "Start" Oct 31 13:02:37.478631 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.478615 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 13:02:37.513732 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.513710 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:37.518263 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.518233 2576 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 13:02:37.521046 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.521032 2576 manager.go:341] "Starting Device Plugin manager" Oct 31 13:02:37.521143 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.521075 2576 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 13:02:37.521143 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.521087 2576 server.go:87] "Starting device plugin registration server" Oct 31 13:02:37.521380 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.521367 2576 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 13:02:37.521445 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.521383 2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 13:02:37.521492 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.521473 2576 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 13:02:37.521569 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.521553 2576 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 13:02:37.521569 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.521567 2576 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 13:02:37.522284 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.522265 2576 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 13:02:37.522358 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.522313 2576 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:37.530653 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.530633 2576 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-128-41.us-east-2.compute.internal\" not found" node="ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.548995 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.548963 2576 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal"] Oct 31 13:02:37.549095 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.549028 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 13:02:37.549967 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.549953 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 13:02:37.550038 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.549979 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 13:02:37.550038 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.549992 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 13:02:37.552250 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.552239 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 13:02:37.552429 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.552414 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.552466 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.552452 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 13:02:37.553035 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.553016 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 13:02:37.553125 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.553044 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 13:02:37.553125 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.553054 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 13:02:37.553125 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.553020 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 13:02:37.553218 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.553133 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 13:02:37.553218 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.553146 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 13:02:37.555136 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.555120 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.555206 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.555150 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 13:02:37.555809 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.555794 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 13:02:37.555876 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.555824 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 13:02:37.555876 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.555838 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 13:02:37.582264 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.582248 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" node="ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.584739 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.584722 2576 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" node="ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.614579 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.614554 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37af5d4f145227f4335887a01dc2c4be-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"37af5d4f145227f4335887a01dc2c4be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.614716 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.614579 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37af5d4f145227f4335887a01dc2c4be-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"37af5d4f145227f4335887a01dc2c4be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.614716 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.614627 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/16434a8e882a2ee35299274e3b318497-config\") pod \"kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"16434a8e882a2ee35299274e3b318497\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.622074 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.622058 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 13:02:37.622962 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.622945 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 13:02:37.623061 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.622982 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 13:02:37.623061 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.622998 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 13:02:37.623061 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.623050 2576 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.631746 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.631729 2576 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.631845 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.631755 2576 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-41.us-east-2.compute.internal\": node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:37.661554 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.661520 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:37.715395 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.715364 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37af5d4f145227f4335887a01dc2c4be-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"37af5d4f145227f4335887a01dc2c4be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.715543 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.715401 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37af5d4f145227f4335887a01dc2c4be-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"37af5d4f145227f4335887a01dc2c4be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.715543 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.715425 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/16434a8e882a2ee35299274e3b318497-config\") pod \"kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"16434a8e882a2ee35299274e3b318497\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.715543 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.715492 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37af5d4f145227f4335887a01dc2c4be-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"37af5d4f145227f4335887a01dc2c4be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.715543 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.715509 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/16434a8e882a2ee35299274e3b318497-config\") pod \"kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"16434a8e882a2ee35299274e3b318497\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.715749 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.715492 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37af5d4f145227f4335887a01dc2c4be-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"37af5d4f145227f4335887a01dc2c4be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.762670 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.762585 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:37.863131 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.863090 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:37.885294 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.885266 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.887431 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:37.887416 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:37.963773 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:37.963742 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:38.064284 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:38.064199 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:38.166377 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:38.166344 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:38.248858 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.248823 2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 31 13:02:38.249310 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.248959 2576 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 31 13:02:38.266576 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:38.266544 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:38.300500 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.300466 2576 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 13:02:38.367401 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:38.367184 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:38.454579 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.454549 2576 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 13:02:38.455564 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.455543 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 13:02:38.455694 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.455575 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 13:02:38.455694 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.455585 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 13:02:38.468155 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:38.468133 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:38.474765 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.474714 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:57:37 +0000 UTC" deadline="2028-06-27 23:31:54.408988415 +0000 UTC" Oct 31 13:02:38.474765 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.474760 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14530h29m15.934232643s" Oct 31 13:02:38.504853 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:38.504809 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16434a8e882a2ee35299274e3b318497.slice/crio-4516dd758e1729253b19122fc3125003c52f4727674a48c15031e664419aee42 WatchSource:0}: Error finding container 4516dd758e1729253b19122fc3125003c52f4727674a48c15031e664419aee42: Status 404 returned error can't find the container with id 4516dd758e1729253b19122fc3125003c52f4727674a48c15031e664419aee42 Oct 31 13:02:38.517752 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.517729 2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:02:38.525735 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:38.525711 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37af5d4f145227f4335887a01dc2c4be.slice/crio-881b5c111023e271fe18bd5f38d10e6785b51aa743839d14c6155c1cef8a4fdb WatchSource:0}: Error finding container 881b5c111023e271fe18bd5f38d10e6785b51aa743839d14c6155c1cef8a4fdb: Status 404 returned error can't find the container with id 881b5c111023e271fe18bd5f38d10e6785b51aa743839d14c6155c1cef8a4fdb Oct 31 13:02:38.570362 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:38.570325 2576 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-41.us-east-2.compute.internal\" not found" Oct 31 13:02:38.610058 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.610030 2576 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 13:02:38.613110 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.613094 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:38.630467 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.630411 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 13:02:38.631505 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.631491 2576 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:02:38.640291 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.640274 2576 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 13:02:38.701815 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:38.701788 2576 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 13:02:39.289290 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.289265 2576 apiserver.go:52] "Watching apiserver" Oct 31 13:02:39.298468 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.298438 2576 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 13:02:39.298930 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.298900 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-ddf7z","openshift-cluster-node-tuning-operator/tuned-qdllj","kube-system/konnectivity-agent-rmt2g","openshift-dns/node-resolver-l5vvf","openshift-multus/network-metrics-daemon-sxmc2","openshift-network-diagnostics/network-check-target-7f7h9","openshift-network-operator/iptables-alerter-zqcpp","openshift-ovn-kubernetes/ovnkube-node-26l6d","kube-system/global-pull-secret-syncer-48bm7","kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf","openshift-image-registry/node-ca-fbstt","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-28gmd"] Oct 31 13:02:39.301605 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.301564 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:02:39.303952 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.303926 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.306595 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.306566 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:02:39.306739 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.306726 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 13:02:39.307035 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.306931 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 13:02:39.307035 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.306987 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 13:02:39.307186 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.307095 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkxw2\"" Oct 31 13:02:39.308074 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.308056 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 13:02:39.308187 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.308164 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 13:02:39.308285 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.308244 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tmkp5\"" Oct 31 13:02:39.308285 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.308282 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 13:02:39.308985 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.308756 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:02:39.311195 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.311175 2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 31 13:02:39.311340 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.311321 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:39.311441 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:39.311411 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:02:39.311837 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.311788 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 13:02:39.313709 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.313687 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:39.313819 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:39.313760 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:02:39.314490 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.314466 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 13:02:39.314873 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.314854 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h72qs\"" Oct 31 13:02:39.315333 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.314967 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 13:02:39.315333 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.315046 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 13:02:39.315704 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.315581 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-65v8h\"" Oct 31 13:02:39.316308 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.316069 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:39.316308 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:39.316132 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:02:39.318674 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.318603 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.321788 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.321772 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.323840 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.323774 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.324576 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.324552 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n54p8\" (UniqueName: \"kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8\") pod \"network-check-target-7f7h9\" (UID: \"018c5e83-597e-4c20-9063-5287b4c332a2\") " pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:39.324698 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.324589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0104a2a9-918a-40bf-9ece-bfa7360a8d36-iptables-alerter-script\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:02:39.324698 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.324631 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm6n8\" (UniqueName: \"kubernetes.io/projected/0104a2a9-918a-40bf-9ece-bfa7360a8d36-kube-api-access-jm6n8\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:02:39.324698 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.324658 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-socket-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.324873 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.324700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:39.324873 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.324754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.324873 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.324816 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-device-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.324873 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.324853 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc1ff283-237c-45f3-a90f-88cb75bcb863-hosts-file\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:02:39.325032 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.324898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cw7j7\" (UniqueName: \"kubernetes.io/projected/70a6eabd-c790-46fa-a9f1-cff156624c2f-kube-api-access-cw7j7\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:39.325032 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.324928 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.325032 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.324953 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-sys-fs\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.325145 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.325032 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0104a2a9-918a-40bf-9ece-bfa7360a8d36-host-slash\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:02:39.325145 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.325072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-registration-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.325145 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.325097 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1-agent-certs\") pod \"konnectivity-agent-rmt2g\" (UID: \"7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1\") " pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:02:39.325335 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.325155 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1-konnectivity-ca\") pod \"konnectivity-agent-rmt2g\" (UID: \"7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1\") " pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:02:39.325335 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.325200 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg99f\" (UniqueName: \"kubernetes.io/projected/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-kube-api-access-rg99f\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.325335 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.325233 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1ff283-237c-45f3-a90f-88cb75bcb863-tmp-dir\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:02:39.325335 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.325286 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr7hd\" (UniqueName: \"kubernetes.io/projected/dc1ff283-237c-45f3-a90f-88cb75bcb863-kube-api-access-nr7hd\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:02:39.326456 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.326422 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.329066 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.329044 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:02:39.335551 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.333061 2576 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 31 13:02:39.383501 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.383466 2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xzpdm" Oct 31 13:02:39.397673 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.397644 2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xzpdm" Oct 31 13:02:39.413947 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.413917 2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 13:02:39.416300 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.416263 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 13:02:39.416300 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.416264 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 13:02:39.416300 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.416268 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 13:02:39.416633 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.416609 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 13:02:39.416955 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.416936 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 13:02:39.417025 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.416996 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 13:02:39.417025 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.417013 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 13:02:39.417125 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.417069 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 13:02:39.417125 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.417093 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 13:02:39.417125 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.417101 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s8mtw\"" Oct 31 13:02:39.417289 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.417129 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 13:02:39.417289 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.417167 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-glfvs\"" Oct 31 13:02:39.417289 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.417185 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r2wwf\"" Oct 31 13:02:39.417289 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.417234 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s9d6g\"" Oct 31 13:02:39.417289 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.417264 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 13:02:39.418151 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.418054 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 13:02:39.418151 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.418109 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 13:02:39.418360 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.418272 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 13:02:39.418411 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.418385 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 13:02:39.418611 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.418513 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 13:02:39.418611 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.418533 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lxhpt\"" Oct 31 13:02:39.418611 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.418580 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 13:02:39.425774 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.425736 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm6n8\" (UniqueName: \"kubernetes.io/projected/0104a2a9-918a-40bf-9ece-bfa7360a8d36-kube-api-access-jm6n8\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:02:39.425881 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.425789 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-system-cni-dir\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.425881 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.425811 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.425881 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.425829 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.425881 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.425846 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysctl-conf\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.425881 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.425870 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-host\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.425904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65x7x\" (UniqueName: \"kubernetes.io/projected/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-kube-api-access-65x7x\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.425924 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-cni-multus\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.425944 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-device-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.425960 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc1ff283-237c-45f3-a90f-88cb75bcb863-hosts-file\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.425976 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-tuned\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.425991 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-systemd\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426020 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-env-overrides\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426036 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2aee367b-5b87-4bfe-9620-5adf75e25b63-dbus\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426053 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-sys\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426068 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr7hd\" (UniqueName: \"kubernetes.io/projected/dc1ff283-237c-45f3-a90f-88cb75bcb863-kube-api-access-nr7hd\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426096 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-etc-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426123 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovn-node-metrics-cert\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426150 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-socket-dir-parent\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.426159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426168 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-daemon-config\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426183 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-cni-binary-copy\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426198 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05baffb8-cba0-444a-989e-dc5df0c7bcd6-tmp\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426213 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq4sr\" (UniqueName: \"kubernetes.io/projected/54a25982-6507-405a-8e10-81d965ab7f4f-kube-api-access-nq4sr\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426242 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-node-log\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426289 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-cnibin\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426215 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc1ff283-237c-45f3-a90f-88cb75bcb863-hosts-file\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426368 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-device-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426409 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-netns\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426439 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426467 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-run-netns\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426494 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-conf-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426529 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-run\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426555 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-cni-binary-copy\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426585 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg99f\" (UniqueName: \"kubernetes.io/projected/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-kube-api-access-rg99f\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426625 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426702 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slrrr\" (UniqueName: \"kubernetes.io/projected/05baffb8-cba0-444a-989e-dc5df0c7bcd6-kube-api-access-slrrr\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.426853 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426732 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-kubelet\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426765 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-kubelet\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426783 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-etc-kubernetes\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426803 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-slash\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426833 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-system-cni-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426868 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx88t\" (UniqueName: \"kubernetes.io/projected/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-kube-api-access-sx88t\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426898 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426915 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-log-socket\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.426959 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427359 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427395 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427431 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0104a2a9-918a-40bf-9ece-bfa7360a8d36-host-slash\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427457 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1-agent-certs\") pod \"konnectivity-agent-rmt2g\" (UID: \"7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1\") " pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427484 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2aee367b-5b87-4bfe-9620-5adf75e25b63-kubelet-config\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427511 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-modprobe-d\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427535 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-systemd\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.427578 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427560 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-cni-bin\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427584 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1ff283-237c-45f3-a90f-88cb75bcb863-tmp-dir\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427607 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-lib-modules\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427632 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-systemd-units\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427657 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-var-lib-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427704 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-cni-netd\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427729 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-cni-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427778 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54p8\" (UniqueName: \"kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8\") pod \"network-check-target-7f7h9\" (UID: \"018c5e83-597e-4c20-9063-5287b4c332a2\") " pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427804 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0104a2a9-918a-40bf-9ece-bfa7360a8d36-iptables-alerter-script\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427829 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-socket-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427854 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427879 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427904 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-cnibin\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427929 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-k8s-cni-cncf-io\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427955 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw7j7\" (UniqueName: \"kubernetes.io/projected/70a6eabd-c790-46fa-a9f1-cff156624c2f-kube-api-access-cw7j7\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.427982 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-kubernetes\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428006 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-var-lib-kubelet\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.428130 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428031 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54a25982-6507-405a-8e10-81d965ab7f4f-host\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428056 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54a25982-6507-405a-8e10-81d965ab7f4f-serviceca\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-hostroot\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428115 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-multus-certs\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428141 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-sys-fs\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428177 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-registration-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428218 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1-konnectivity-ca\") pod \"konnectivity-agent-rmt2g\" (UID: \"7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1\") " pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428247 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v79lg\" (UniqueName: \"kubernetes.io/projected/19180e90-ffed-457b-9d2e-92645efe3491-kube-api-access-v79lg\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428274 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysconfig\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428299 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysctl-d\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428326 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-ovn\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428352 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovnkube-config\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428377 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovnkube-script-lib\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428403 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-cni-bin\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428609 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.428856 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.428681 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0104a2a9-918a-40bf-9ece-bfa7360a8d36-host-slash\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:02:39.429576 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.429040 2576 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 13:02:39.429766 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.429737 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1ff283-237c-45f3-a90f-88cb75bcb863-tmp-dir\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:02:39.430283 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.430260 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-sys-fs\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.430283 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:39.430272 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:02:39.430451 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.430300 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-registration-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.430451 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:39.430386 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs podName:70a6eabd-c790-46fa-a9f1-cff156624c2f nodeName:}" failed. No retries permitted until 2026-10-31 13:02:39.930348849 +0000 UTC m=+2.993146237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs") pod "network-metrics-daemon-sxmc2" (UID: "70a6eabd-c790-46fa-a9f1-cff156624c2f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:02:39.431157 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.431133 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0104a2a9-918a-40bf-9ece-bfa7360a8d36-iptables-alerter-script\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:02:39.431157 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.431147 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1-konnectivity-ca\") pod \"konnectivity-agent-rmt2g\" (UID: \"7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1\") " pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:02:39.431264 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.431220 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-socket-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.434244 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.434217 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1-agent-certs\") pod \"konnectivity-agent-rmt2g\" (UID: \"7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1\") " pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:02:39.441959 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.441930 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr7hd\" (UniqueName: \"kubernetes.io/projected/dc1ff283-237c-45f3-a90f-88cb75bcb863-kube-api-access-nr7hd\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:02:39.442137 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.442115 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg99f\" (UniqueName: \"kubernetes.io/projected/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-kube-api-access-rg99f\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.442282 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.442261 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm6n8\" (UniqueName: \"kubernetes.io/projected/0104a2a9-918a-40bf-9ece-bfa7360a8d36-kube-api-access-jm6n8\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:02:39.448500 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:39.448479 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 13:02:39.448607 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:39.448508 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 13:02:39.448607 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:39.448523 2576 projected.go:196] Error preparing data for projected volume kube-api-access-n54p8 for pod openshift-network-diagnostics/network-check-target-7f7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:02:39.448607 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:39.448586 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8 podName:018c5e83-597e-4c20-9063-5287b4c332a2 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:39.948566557 +0000 UTC m=+3.011363938 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-n54p8" (UniqueName: "kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8") pod "network-check-target-7f7h9" (UID: "018c5e83-597e-4c20-9063-5287b4c332a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:02:39.451745 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.451718 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw7j7\" (UniqueName: \"kubernetes.io/projected/70a6eabd-c790-46fa-a9f1-cff156624c2f-kube-api-access-cw7j7\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:39.454072 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.453988 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" event={"ID":"16434a8e882a2ee35299274e3b318497","Type":"ContainerStarted","Data":"4516dd758e1729253b19122fc3125003c52f4727674a48c15031e664419aee42"} Oct 31 13:02:39.455700 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.455671 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" event={"ID":"37af5d4f145227f4335887a01dc2c4be","Type":"ContainerStarted","Data":"881b5c111023e271fe18bd5f38d10e6785b51aa743839d14c6155c1cef8a4fdb"} Oct 31 13:02:39.528913 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.528872 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2aee367b-5b87-4bfe-9620-5adf75e25b63-kubelet-config\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:39.529095 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.528921 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-modprobe-d\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.529095 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.528956 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-systemd\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.529095 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.528984 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-cni-bin\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.529095 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529013 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-lib-modules\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.529095 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529035 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-systemd-units\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.529095 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529059 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-var-lib-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.529095 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529092 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-cni-netd\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.529397 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-cni-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.529397 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529197 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:39.529397 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529229 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-cnibin\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.529397 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529268 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-k8s-cni-cncf-io\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.529397 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-kubernetes\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.529397 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529324 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-var-lib-kubelet\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.529397 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54a25982-6507-405a-8e10-81d965ab7f4f-host\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:02:39.529397 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529369 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54a25982-6507-405a-8e10-81d965ab7f4f-serviceca\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:02:39.529397 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529393 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-hostroot\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.529811 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529418 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-multus-certs\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.529811 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529456 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79lg\" (UniqueName: \"kubernetes.io/projected/19180e90-ffed-457b-9d2e-92645efe3491-kube-api-access-v79lg\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.529811 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysconfig\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.529811 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529512 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysctl-d\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.529811 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529536 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-ovn\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.529811 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529561 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovnkube-config\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.529811 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529631 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovnkube-script-lib\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.529811 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529686 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-cni-bin\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.529811 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529783 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-system-cni-dir\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529828 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529852 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529874 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysctl-conf\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529903 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-host\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529918 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65x7x\" (UniqueName: \"kubernetes.io/projected/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-kube-api-access-65x7x\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529940 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-cni-multus\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.529962 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-tuned\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530009 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-systemd\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530067 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-env-overrides\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530101 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2aee367b-5b87-4bfe-9620-5adf75e25b63-dbus\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530144 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-sys\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530162 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-etc-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530178 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovn-node-metrics-cert\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.530247 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530193 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-socket-dir-parent\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530272 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-daemon-config\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530289 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-cni-binary-copy\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530314 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05baffb8-cba0-444a-989e-dc5df0c7bcd6-tmp\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530349 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4sr\" (UniqueName: \"kubernetes.io/projected/54a25982-6507-405a-8e10-81d965ab7f4f-kube-api-access-nq4sr\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530377 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-node-log\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530405 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-cnibin\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-netns\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530461 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-run-netns\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530485 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-conf-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530509 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-run\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530541 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-cni-binary-copy\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530572 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slrrr\" (UniqueName: \"kubernetes.io/projected/05baffb8-cba0-444a-989e-dc5df0c7bcd6-kube-api-access-slrrr\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530602 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-kubelet\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530629 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-kubelet\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530693 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-etc-kubernetes\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530742 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-slash\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.530829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530822 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-system-cni-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.531600 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530847 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx88t\" (UniqueName: \"kubernetes.io/projected/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-kube-api-access-sx88t\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.531600 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530894 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.531600 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530915 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-log-socket\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.531600 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530950 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.531600 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.530981 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.531600 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.531100 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.531600 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.531171 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2aee367b-5b87-4bfe-9620-5adf75e25b63-kubelet-config\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:39.531600 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.531266 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-modprobe-d\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.531600 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.531307 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-systemd\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.532056 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.531795 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-cni-bin\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.532056 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532016 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-lib-modules\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.532162 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532085 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-systemd-units\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.532162 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532137 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-var-lib-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.532264 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532188 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-cni-netd\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.532318 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532310 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-cni-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:39.532425 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532467 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-conf-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532532 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-k8s-cni-cncf-io\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532553 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-cnibin\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:39.532542 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret podName:2aee367b-5b87-4bfe-9620-5adf75e25b63 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:40.032502361 +0000 UTC m=+3.095299728 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret") pod "global-pull-secret-syncer-48bm7" (UID: "2aee367b-5b87-4bfe-9620-5adf75e25b63") : object "kube-system"/"original-pull-secret" not registered Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532633 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-var-lib-kubelet\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532642 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-sys\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532730 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-hostroot\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532798 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-multus-certs\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532955 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-run\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.533267 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-cnibin\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.533500 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysctl-d\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.533560 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-ovn\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.533719 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-cni-binary-copy\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.533752 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-node-log\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.533817 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54a25982-6507-405a-8e10-81d965ab7f4f-host\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.533837 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysconfig\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.536073 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.534399 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-host\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.534710 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-kubelet\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.534780 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-kubelet\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.534837 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-etc-kubernetes\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.534955 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovnkube-config\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.535053 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.535154 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-log-socket\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.535167 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-system-cni-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.535334 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovnkube-script-lib\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.532466 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2aee367b-5b87-4bfe-9620-5adf75e25b63-dbus\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.535926 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-slash\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.535930 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-netns\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.535944 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-systemd\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.535985 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.536037 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-kubernetes\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.536041 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysctl-conf\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.536317 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.536563 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54a25982-6507-405a-8e10-81d965ab7f4f-serviceca\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:02:39.537211 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.536630 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-system-cni-dir\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.538082 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.536736 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-socket-dir-parent\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.538082 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.536737 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-cni-multus\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.538082 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.536781 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-run-netns\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.538082 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.536813 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.538082 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.536834 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-etc-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.538082 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.537268 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-env-overrides\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.538082 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.537337 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-cni-bin\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.538082 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.537340 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-daemon-config\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.538082 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.537466 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-cni-binary-copy\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.538082 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.537487 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.540356 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.540284 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovn-node-metrics-cert\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.540492 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.540429 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05baffb8-cba0-444a-989e-dc5df0c7bcd6-tmp\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.540758 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.540734 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-tuned\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.550357 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.550076 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79lg\" (UniqueName: \"kubernetes.io/projected/19180e90-ffed-457b-9d2e-92645efe3491-kube-api-access-v79lg\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.553877 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.553848 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65x7x\" (UniqueName: \"kubernetes.io/projected/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-kube-api-access-65x7x\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.555520 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.555492 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slrrr\" (UniqueName: \"kubernetes.io/projected/05baffb8-cba0-444a-989e-dc5df0c7bcd6-kube-api-access-slrrr\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.557747 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.557701 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq4sr\" (UniqueName: \"kubernetes.io/projected/54a25982-6507-405a-8e10-81d965ab7f4f-kube-api-access-nq4sr\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:02:39.559150 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.559125 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx88t\" (UniqueName: \"kubernetes.io/projected/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-kube-api-access-sx88t\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.619860 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.619824 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:02:39.630583 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.630559 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:02:39.631486 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:39.631452 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0104a2a9_918a_40bf_9ece_bfa7360a8d36.slice/crio-270e54d1580e36f820e948bcc22a171e5adfbd5673f865b2c8897cf6859ff825 WatchSource:0}: Error finding container 270e54d1580e36f820e948bcc22a171e5adfbd5673f865b2c8897cf6859ff825: Status 404 returned error can't find the container with id 270e54d1580e36f820e948bcc22a171e5adfbd5673f865b2c8897cf6859ff825 Oct 31 13:02:39.642173 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:39.642144 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d41cf67_3992_4b1f_8d1d_4e7bd87a01e1.slice/crio-ed98c2b2756f3de31628d4bf91d8ca9a7314abd9d9b3018841dee14b910e30a2 WatchSource:0}: Error finding container ed98c2b2756f3de31628d4bf91d8ca9a7314abd9d9b3018841dee14b910e30a2: Status 404 returned error can't find the container with id ed98c2b2756f3de31628d4bf91d8ca9a7314abd9d9b3018841dee14b910e30a2 Oct 31 13:02:39.644095 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.644068 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:02:39.648102 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.648058 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:02:39.656551 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:39.656517 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode45e4264_8ba9_4ec4_be49_ccc8d26472f1.slice/crio-d2078686775b9649bc6bc06bb416b4930cdc01332e3a585abd56a575d299d979 WatchSource:0}: Error finding container d2078686775b9649bc6bc06bb416b4930cdc01332e3a585abd56a575d299d979: Status 404 returned error can't find the container with id d2078686775b9649bc6bc06bb416b4930cdc01332e3a585abd56a575d299d979 Oct 31 13:02:39.658971 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:39.658934 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1ff283_237c_45f3_a90f_88cb75bcb863.slice/crio-8c6b18eec02cd699f5ce009b11c8d6926a0fb8f98e3883d83d5bea81d45f9abd WatchSource:0}: Error finding container 8c6b18eec02cd699f5ce009b11c8d6926a0fb8f98e3883d83d5bea81d45f9abd: Status 404 returned error can't find the container with id 8c6b18eec02cd699f5ce009b11c8d6926a0fb8f98e3883d83d5bea81d45f9abd Oct 31 13:02:39.671162 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.671137 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:02:39.677708 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.677684 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:02:39.683393 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:39.683366 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05baffb8_cba0_444a_989e_dc5df0c7bcd6.slice/crio-ac9c12a8389d8ba680e8a73422fd23147d3671ada78b0f889aaa6714396842a9 WatchSource:0}: Error finding container ac9c12a8389d8ba680e8a73422fd23147d3671ada78b0f889aaa6714396842a9: Status 404 returned error can't find the container with id ac9c12a8389d8ba680e8a73422fd23147d3671ada78b0f889aaa6714396842a9 Oct 31 13:02:39.688791 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.688773 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-ddf7z" Oct 31 13:02:39.690500 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:39.690472 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19180e90_ffed_457b_9d2e_92645efe3491.slice/crio-9eedaf1a1fe55c6516bdcc356943d841e84ed7bc35fb6208aa3f7aead2eb62c2 WatchSource:0}: Error finding container 9eedaf1a1fe55c6516bdcc356943d841e84ed7bc35fb6208aa3f7aead2eb62c2: Status 404 returned error can't find the container with id 9eedaf1a1fe55c6516bdcc356943d841e84ed7bc35fb6208aa3f7aead2eb62c2 Oct 31 13:02:39.691167 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.691148 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:02:39.700843 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.700826 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:02:39.701364 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:39.701336 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81aa31ec_a7a8_4ce8_9097_1c3cec6b49c5.slice/crio-48e0fc447fee8b31eb4bf5419ebf18b2d26563d1df3528bbe6eeba897a599d65 WatchSource:0}: Error finding container 48e0fc447fee8b31eb4bf5419ebf18b2d26563d1df3528bbe6eeba897a599d65: Status 404 returned error can't find the container with id 48e0fc447fee8b31eb4bf5419ebf18b2d26563d1df3528bbe6eeba897a599d65 Oct 31 13:02:39.703494 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:39.703471 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a25982_6507_405a_8e10_81d965ab7f4f.slice/crio-89f3a55866b70c818d5c596220497aa026d37c6c0afe9c19d4852c7771fb6673 WatchSource:0}: Error finding container 89f3a55866b70c818d5c596220497aa026d37c6c0afe9c19d4852c7771fb6673: Status 404 returned error can't find the container with id 89f3a55866b70c818d5c596220497aa026d37c6c0afe9c19d4852c7771fb6673 Oct 31 13:02:39.713374 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:02:39.713343 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2eab6ed_c191_4b42_899c_e816ab8d4a0b.slice/crio-4391b5b81d24ce2493d2f0b7b9b6ef23d8f563c80d320aa9092dd8c3fe4f09ed WatchSource:0}: Error finding container 4391b5b81d24ce2493d2f0b7b9b6ef23d8f563c80d320aa9092dd8c3fe4f09ed: Status 404 returned error can't find the container with id 4391b5b81d24ce2493d2f0b7b9b6ef23d8f563c80d320aa9092dd8c3fe4f09ed Oct 31 13:02:39.798904 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.798814 2576 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 13:02:39.934046 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:39.934010 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:39.934251 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:39.934227 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:02:39.934343 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:39.934303 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs podName:70a6eabd-c790-46fa-a9f1-cff156624c2f nodeName:}" failed. No retries permitted until 2026-10-31 13:02:40.934288624 +0000 UTC m=+3.997086009 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs") pod "network-metrics-daemon-sxmc2" (UID: "70a6eabd-c790-46fa-a9f1-cff156624c2f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:02:40.035322 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.035293 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54p8\" (UniqueName: \"kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8\") pod \"network-check-target-7f7h9\" (UID: \"018c5e83-597e-4c20-9063-5287b4c332a2\") " pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:40.035432 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.035340 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:40.035534 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:40.035459 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 13:02:40.035534 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:40.035475 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 13:02:40.035534 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:40.035492 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 13:02:40.035534 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:40.035505 2576 projected.go:196] Error preparing data for projected volume kube-api-access-n54p8 for pod openshift-network-diagnostics/network-check-target-7f7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:02:40.035534 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:40.035529 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret podName:2aee367b-5b87-4bfe-9620-5adf75e25b63 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:41.035510267 +0000 UTC m=+4.098307634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret") pod "global-pull-secret-syncer-48bm7" (UID: "2aee367b-5b87-4bfe-9620-5adf75e25b63") : object "kube-system"/"original-pull-secret" not registered Oct 31 13:02:40.035826 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:40.035549 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8 podName:018c5e83-597e-4c20-9063-5287b4c332a2 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:41.035540568 +0000 UTC m=+4.098337939 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-n54p8" (UniqueName: "kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8") pod "network-check-target-7f7h9" (UID: "018c5e83-597e-4c20-9063-5287b4c332a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:02:40.402094 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.401981 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:57:39 +0000 UTC" deadline="2028-04-13 18:11:36.73991378 +0000 UTC" Oct 31 13:02:40.402094 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.402091 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12725h8m56.337827415s" Oct 31 13:02:40.449785 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.449753 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:40.449962 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.449753 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:40.449962 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:40.449895 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:02:40.450042 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:40.449974 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:02:40.463811 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.463761 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rmt2g" event={"ID":"7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1","Type":"ContainerStarted","Data":"ed98c2b2756f3de31628d4bf91d8ca9a7314abd9d9b3018841dee14b910e30a2"} Oct 31 13:02:40.475374 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.475322 2576 generic.go:356] "Generic (PLEG): container finished" podID="37af5d4f145227f4335887a01dc2c4be" containerID="fd171edb3552762fd6f448c3835e83bb5820186d2707cc25302ee1fdec900077" exitCode=0 Oct 31 13:02:40.475462 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.475430 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" event={"ID":"37af5d4f145227f4335887a01dc2c4be","Type":"ContainerDied","Data":"fd171edb3552762fd6f448c3835e83bb5820186d2707cc25302ee1fdec900077"} Oct 31 13:02:40.493044 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.492958 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ddf7z" event={"ID":"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5","Type":"ContainerStarted","Data":"48e0fc447fee8b31eb4bf5419ebf18b2d26563d1df3528bbe6eeba897a599d65"} Oct 31 13:02:40.502744 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.502713 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28gmd" event={"ID":"19180e90-ffed-457b-9d2e-92645efe3491","Type":"ContainerStarted","Data":"9eedaf1a1fe55c6516bdcc356943d841e84ed7bc35fb6208aa3f7aead2eb62c2"} Oct 31 13:02:40.511660 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.511373 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qdllj" event={"ID":"05baffb8-cba0-444a-989e-dc5df0c7bcd6","Type":"ContainerStarted","Data":"ac9c12a8389d8ba680e8a73422fd23147d3671ada78b0f889aaa6714396842a9"} Oct 31 13:02:40.521042 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.520922 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l5vvf" event={"ID":"dc1ff283-237c-45f3-a90f-88cb75bcb863","Type":"ContainerStarted","Data":"8c6b18eec02cd699f5ce009b11c8d6926a0fb8f98e3883d83d5bea81d45f9abd"} Oct 31 13:02:40.532773 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.532741 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zqcpp" event={"ID":"0104a2a9-918a-40bf-9ece-bfa7360a8d36","Type":"ContainerStarted","Data":"270e54d1580e36f820e948bcc22a171e5adfbd5673f865b2c8897cf6859ff825"} Oct 31 13:02:40.535102 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.535073 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" event={"ID":"d2eab6ed-c191-4b42-899c-e816ab8d4a0b","Type":"ContainerStarted","Data":"4391b5b81d24ce2493d2f0b7b9b6ef23d8f563c80d320aa9092dd8c3fe4f09ed"} Oct 31 13:02:40.546463 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.546369 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fbstt" event={"ID":"54a25982-6507-405a-8e10-81d965ab7f4f","Type":"ContainerStarted","Data":"89f3a55866b70c818d5c596220497aa026d37c6c0afe9c19d4852c7771fb6673"} Oct 31 13:02:40.557437 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.557361 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" event={"ID":"e45e4264-8ba9-4ec4-be49-ccc8d26472f1","Type":"ContainerStarted","Data":"d2078686775b9649bc6bc06bb416b4930cdc01332e3a585abd56a575d299d979"} Oct 31 13:02:40.947922 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:40.947031 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:40.947922 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:40.947487 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:02:40.947922 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:40.947557 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs podName:70a6eabd-c790-46fa-a9f1-cff156624c2f nodeName:}" failed. No retries permitted until 2026-10-31 13:02:42.947538066 +0000 UTC m=+6.010335446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs") pod "network-metrics-daemon-sxmc2" (UID: "70a6eabd-c790-46fa-a9f1-cff156624c2f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:02:41.047968 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:41.047876 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54p8\" (UniqueName: \"kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8\") pod \"network-check-target-7f7h9\" (UID: \"018c5e83-597e-4c20-9063-5287b4c332a2\") " pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:41.047968 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:41.047945 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:41.048209 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:41.048104 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 13:02:41.048209 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:41.048168 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret podName:2aee367b-5b87-4bfe-9620-5adf75e25b63 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:43.048149324 +0000 UTC m=+6.110946713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret") pod "global-pull-secret-syncer-48bm7" (UID: "2aee367b-5b87-4bfe-9620-5adf75e25b63") : object "kube-system"/"original-pull-secret" not registered Oct 31 13:02:41.048654 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:41.048630 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 13:02:41.048804 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:41.048660 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 13:02:41.048804 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:41.048689 2576 projected.go:196] Error preparing data for projected volume kube-api-access-n54p8 for pod openshift-network-diagnostics/network-check-target-7f7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:02:41.048804 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:41.048741 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8 podName:018c5e83-597e-4c20-9063-5287b4c332a2 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:43.048724591 +0000 UTC m=+6.111521963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-n54p8" (UniqueName: "kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8") pod "network-check-target-7f7h9" (UID: "018c5e83-597e-4c20-9063-5287b4c332a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:02:41.403271 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:41.403071 2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:57:39 +0000 UTC" deadline="2028-05-18 09:12:21.106875173 +0000 UTC" Oct 31 13:02:41.403271 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:41.403124 2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13556h9m39.70375435s" Oct 31 13:02:41.450955 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:41.450918 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:41.451132 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:41.451072 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:02:41.569455 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:41.569368 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" event={"ID":"37af5d4f145227f4335887a01dc2c4be","Type":"ContainerStarted","Data":"a911dcce01965c0acbfb6569bb22eb1645f6adb2b13f745f2765530f92f14bde"} Oct 31 13:02:41.577084 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:41.576536 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" event={"ID":"16434a8e882a2ee35299274e3b318497","Type":"ContainerStarted","Data":"965a2eb5235865b08b70aecb0a83643fdaca11c2141d0dd86f548b380973a52c"} Oct 31 13:02:41.611639 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:41.611574 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" podStartSLOduration=2.611555611 podStartE2EDuration="2.611555611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:41.610882054 +0000 UTC m=+4.673679469" watchObservedRunningTime="2026-10-31 13:02:41.611555611 +0000 UTC m=+4.674353005" Oct 31 13:02:41.611979 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:41.611854 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" podStartSLOduration=2.611848425 podStartE2EDuration="2.611848425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:41.591116014 +0000 UTC m=+4.653913403" watchObservedRunningTime="2026-10-31 13:02:41.611848425 +0000 UTC m=+4.674645818" Oct 31 13:02:42.449903 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:42.449868 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:42.450456 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:42.450023 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:02:42.450456 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:42.450395 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:42.450585 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:42.450541 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:02:42.974776 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:42.974735 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:42.974962 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:42.974904 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:02:42.975030 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:42.974977 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs podName:70a6eabd-c790-46fa-a9f1-cff156624c2f nodeName:}" failed. No retries permitted until 2026-10-31 13:02:46.974956319 +0000 UTC m=+10.037753702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs") pod "network-metrics-daemon-sxmc2" (UID: "70a6eabd-c790-46fa-a9f1-cff156624c2f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:02:43.075864 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:43.075825 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54p8\" (UniqueName: \"kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8\") pod \"network-check-target-7f7h9\" (UID: \"018c5e83-597e-4c20-9063-5287b4c332a2\") " pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:43.076060 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:43.075894 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:43.076060 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:43.076013 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 13:02:43.076060 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:43.076030 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 13:02:43.076060 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:43.076046 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 13:02:43.076060 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:43.076059 2576 projected.go:196] Error preparing data for projected volume kube-api-access-n54p8 for pod openshift-network-diagnostics/network-check-target-7f7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:02:43.076364 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:43.076096 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret podName:2aee367b-5b87-4bfe-9620-5adf75e25b63 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:47.076076293 +0000 UTC m=+10.138873676 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret") pod "global-pull-secret-syncer-48bm7" (UID: "2aee367b-5b87-4bfe-9620-5adf75e25b63") : object "kube-system"/"original-pull-secret" not registered Oct 31 13:02:43.076364 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:43.076117 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8 podName:018c5e83-597e-4c20-9063-5287b4c332a2 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:47.076106129 +0000 UTC m=+10.138903511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-n54p8" (UniqueName: "kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8") pod "network-check-target-7f7h9" (UID: "018c5e83-597e-4c20-9063-5287b4c332a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:02:43.452462 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:43.451921 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:43.452462 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:43.452065 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:02:43.602821 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:43.602777 2576 generic.go:356] "Generic (PLEG): container finished" podID="19180e90-ffed-457b-9d2e-92645efe3491" containerID="050275a9c26b105282f2c96a45c577295c475e97337419ca3f74abbb25173396" exitCode=0 Oct 31 13:02:43.603043 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:43.602845 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28gmd" event={"ID":"19180e90-ffed-457b-9d2e-92645efe3491","Type":"ContainerDied","Data":"050275a9c26b105282f2c96a45c577295c475e97337419ca3f74abbb25173396"} Oct 31 13:02:44.449160 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:44.449076 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:44.449332 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:44.449227 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:02:44.459921 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:44.459829 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:44.460455 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:44.460057 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:02:44.611067 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:44.610222 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rmt2g" event={"ID":"7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1","Type":"ContainerStarted","Data":"ea3876af2d1610546d455c477f5508061b0d884c2fae6427c3c3a8a34989d0b2"} Oct 31 13:02:45.449647 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:45.449540 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:45.449798 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:45.449698 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:02:45.618411 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:45.618365 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fbstt" event={"ID":"54a25982-6507-405a-8e10-81d965ab7f4f","Type":"ContainerStarted","Data":"4532ad90f2388a9f152123fd95f13249d057945fb7f35a39e332f6d528aae27c"} Oct 31 13:02:45.636937 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:45.636171 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rmt2g" podStartSLOduration=3.264554448 podStartE2EDuration="7.63614997s" totalImagesPullingTime="4.371595522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:44.633739396 +0000 UTC m=+7.696536784" watchObservedRunningTime="2026-10-31 13:02:45.63614997 +0000 UTC m=+8.698947357" Oct 31 13:02:45.734292 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:45.734208 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:02:45.735261 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:45.735092 2576 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:02:45.754405 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:45.753631 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fbstt" podStartSLOduration=2.331971803 podStartE2EDuration="7.753613061s" totalImagesPullingTime="5.421641258s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:45.63691339 +0000 UTC m=+8.699710776" watchObservedRunningTime="2026-10-31 13:02:45.753613061 +0000 UTC m=+8.816410442" Oct 31 13:02:46.450403 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:46.449608 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:46.450403 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:46.449747 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:02:46.450403 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:46.450155 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:46.450403 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:46.450276 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:02:46.624937 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:46.624897 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" event={"ID":"e45e4264-8ba9-4ec4-be49-ccc8d26472f1","Type":"ContainerStarted","Data":"29d9a41cd72faa98bf847d27324b388c3ff42e6cac29670db6f392e5cd7efb72"} Oct 31 13:02:47.013398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:47.013361 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:47.013585 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:47.013551 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:02:47.013686 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:47.013635 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs podName:70a6eabd-c790-46fa-a9f1-cff156624c2f nodeName:}" failed. No retries permitted until 2026-10-31 13:02:55.013617154 +0000 UTC m=+18.076414533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs") pod "network-metrics-daemon-sxmc2" (UID: "70a6eabd-c790-46fa-a9f1-cff156624c2f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:02:47.114651 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:47.114609 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54p8\" (UniqueName: \"kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8\") pod \"network-check-target-7f7h9\" (UID: \"018c5e83-597e-4c20-9063-5287b4c332a2\") " pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:47.114803 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:47.114676 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:47.114840 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:47.114823 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 13:02:47.114899 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:47.114885 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret podName:2aee367b-5b87-4bfe-9620-5adf75e25b63 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:55.114866057 +0000 UTC m=+18.177663446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret") pod "global-pull-secret-syncer-48bm7" (UID: "2aee367b-5b87-4bfe-9620-5adf75e25b63") : object "kube-system"/"original-pull-secret" not registered Oct 31 13:02:47.115334 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:47.115312 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 13:02:47.115411 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:47.115337 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 13:02:47.115411 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:47.115351 2576 projected.go:196] Error preparing data for projected volume kube-api-access-n54p8 for pod openshift-network-diagnostics/network-check-target-7f7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:02:47.115411 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:47.115397 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8 podName:018c5e83-597e-4c20-9063-5287b4c332a2 nodeName:}" failed. No retries permitted until 2026-10-31 13:02:55.115382306 +0000 UTC m=+18.178179672 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-n54p8" (UniqueName: "kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8") pod "network-check-target-7f7h9" (UID: "018c5e83-597e-4c20-9063-5287b4c332a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:02:47.451292 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:47.449876 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:47.451292 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:47.449971 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:02:47.470825 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:47.470781 2576 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 13:02:47.524857 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:47.524730 2576 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T13:02:47.470801553Z","UUID":"51b37ef8-18a4-4be1-9a37-89de4297d587","Handler":null,"Name":"","Endpoint":""} Oct 31 13:02:47.527820 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:47.527304 2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 13:02:47.527820 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:47.527334 2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 13:02:47.629367 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:47.629256 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" event={"ID":"e45e4264-8ba9-4ec4-be49-ccc8d26472f1","Type":"ContainerStarted","Data":"cb7b38506e01a620c69ee590271c3418f52ddda6d9882ce6f91f476b4eed50d3"} Oct 31 13:02:47.631259 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:47.631234 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 13:02:47.631787 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:47.631750 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-ddf7z" event={"ID":"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5","Type":"ContainerStarted","Data":"ac3f15621f5bdfa7fa3309a2e3573639fe366877db4f9be597d685f66861ea0c"} Oct 31 13:02:47.653985 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:47.653927 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-ddf7z" podStartSLOduration=1.874038712 podStartE2EDuration="9.653908869s" totalImagesPullingTime="7.779870157s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:47.653542337 +0000 UTC m=+10.716339723" watchObservedRunningTime="2026-10-31 13:02:47.653908869 +0000 UTC m=+10.716706259" Oct 31 13:02:48.449562 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:48.449485 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:48.449723 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:48.449643 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:02:48.449786 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:48.449714 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:48.449893 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:48.449864 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:02:48.635064 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:48.634862 2576 generic.go:356] "Generic (PLEG): container finished" podID="19180e90-ffed-457b-9d2e-92645efe3491" containerID="61121f6a198dd263800eacd20102bf9aa75af91b49bf6f9f59ea3201e535b5a1" exitCode=0 Oct 31 13:02:48.635542 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:48.634939 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28gmd" event={"ID":"19180e90-ffed-457b-9d2e-92645efe3491","Type":"ContainerDied","Data":"61121f6a198dd263800eacd20102bf9aa75af91b49bf6f9f59ea3201e535b5a1"} Oct 31 13:02:48.636881 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:48.636831 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l5vvf" event={"ID":"dc1ff283-237c-45f3-a90f-88cb75bcb863","Type":"ContainerStarted","Data":"232f74d1ae7dd0e90404d2d8d0324bb76bc95cffec15d1bd402327bb94f9924f"} Oct 31 13:02:48.678720 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:48.678673 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-l5vvf" podStartSLOduration=2.484696418 podStartE2EDuration="10.678659459s" totalImagesPullingTime="8.193963041s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:48.678276807 +0000 UTC m=+11.741074189" watchObservedRunningTime="2026-10-31 13:02:48.678659459 +0000 UTC m=+11.741456844" Oct 31 13:02:49.449758 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:49.449729 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:49.449874 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:49.449844 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:02:49.640711 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:49.640675 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" event={"ID":"e45e4264-8ba9-4ec4-be49-ccc8d26472f1","Type":"ContainerStarted","Data":"2f279c6757827f1af082cbba6b0ca5205d0ec4273e97219099eb6a9792ee32fa"} Oct 31 13:02:49.642723 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:49.642692 2576 generic.go:356] "Generic (PLEG): container finished" podID="19180e90-ffed-457b-9d2e-92645efe3491" containerID="ee6374eb8ef727cd0ea9f58a0ff6e5f3ba6ff327cd51a7a6058afefafa4936d7" exitCode=0 Oct 31 13:02:49.642850 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:49.642767 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28gmd" event={"ID":"19180e90-ffed-457b-9d2e-92645efe3491","Type":"ContainerDied","Data":"ee6374eb8ef727cd0ea9f58a0ff6e5f3ba6ff327cd51a7a6058afefafa4936d7"} Oct 31 13:02:49.644299 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:49.644274 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-qdllj" event={"ID":"05baffb8-cba0-444a-989e-dc5df0c7bcd6","Type":"ContainerStarted","Data":"00f0aa446923c522c8895f871d3a0f3eeef21b6f8dc30daebe8a6ec680fae20f"} Oct 31 13:02:49.645895 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:49.645878 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zqcpp" event={"ID":"0104a2a9-918a-40bf-9ece-bfa7360a8d36","Type":"ContainerStarted","Data":"675f04ff68e1073aac728135bde6056f98ae660e832221c00b07ebea1f3fa82e"} Oct 31 13:02:49.663676 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:49.663583 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" podStartSLOduration=3.187463235 podStartE2EDuration="11.663572243s" totalImagesPullingTime="8.476109008s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:49.663316262 +0000 UTC m=+12.726113647" watchObservedRunningTime="2026-10-31 13:02:49.663572243 +0000 UTC m=+12.726369628" Oct 31 13:02:49.684221 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:49.684176 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-qdllj" podStartSLOduration=1.9042623170000001 podStartE2EDuration="11.684165086s" totalImagesPullingTime="9.779902769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:49.68369853 +0000 UTC m=+12.746495914" watchObservedRunningTime="2026-10-31 13:02:49.684165086 +0000 UTC m=+12.746962470" Oct 31 13:02:49.701455 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:49.701404 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zqcpp" podStartSLOduration=3.627946685 podStartE2EDuration="11.701390952s" totalImagesPullingTime="8.073444267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:49.700879384 +0000 UTC m=+12.763676770" watchObservedRunningTime="2026-10-31 13:02:49.701390952 +0000 UTC m=+12.764188336" Oct 31 13:02:50.449216 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:50.449180 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:50.449216 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:50.449205 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:50.449390 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:50.449303 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:02:50.449443 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:50.449376 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:02:50.650401 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:50.650324 2576 generic.go:356] "Generic (PLEG): container finished" podID="19180e90-ffed-457b-9d2e-92645efe3491" containerID="ab986b76768bf9f682c15b5d68938b4bfded14614a4cb257c6e1cf323033b377" exitCode=0 Oct 31 13:02:50.650863 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:50.650427 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28gmd" event={"ID":"19180e90-ffed-457b-9d2e-92645efe3491","Type":"ContainerDied","Data":"ab986b76768bf9f682c15b5d68938b4bfded14614a4cb257c6e1cf323033b377"} Oct 31 13:02:51.448874 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:51.448841 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:51.449040 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:51.448971 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:02:51.638624 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:51.638577 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:02:51.638812 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:51.638734 2576 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 13:02:51.639246 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:51.639229 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:02:52.449257 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:52.449232 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:52.449257 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:52.449260 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:52.449870 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:52.449335 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:02:52.449870 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:52.449455 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:02:52.656557 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:52.656529 2576 generic.go:356] "Generic (PLEG): container finished" podID="19180e90-ffed-457b-9d2e-92645efe3491" containerID="6131f117fa7d721279f822017d53dad5a9cb9233686ca6069e3a74465f0a11fd" exitCode=0 Oct 31 13:02:52.656695 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:52.656578 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28gmd" event={"ID":"19180e90-ffed-457b-9d2e-92645efe3491","Type":"ContainerDied","Data":"6131f117fa7d721279f822017d53dad5a9cb9233686ca6069e3a74465f0a11fd"} Oct 31 13:02:53.452803 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:53.452562 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:53.453325 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:53.452888 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:02:53.661770 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:53.661737 2576 generic.go:356] "Generic (PLEG): container finished" podID="19180e90-ffed-457b-9d2e-92645efe3491" containerID="ab3aff25a31ee52e832322a93132b3b4bb25913ee1909a8f90b03872bed4fa01" exitCode=0 Oct 31 13:02:53.661954 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:53.661794 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28gmd" event={"ID":"19180e90-ffed-457b-9d2e-92645efe3491","Type":"ContainerDied","Data":"ab3aff25a31ee52e832322a93132b3b4bb25913ee1909a8f90b03872bed4fa01"} Oct 31 13:02:54.449358 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:54.449326 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:54.449358 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:54.449356 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:54.449633 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:54.449427 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:02:54.449753 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:54.449727 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:02:54.668226 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:54.668189 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-28gmd" event={"ID":"19180e90-ffed-457b-9d2e-92645efe3491","Type":"ContainerStarted","Data":"e60aaad74617be1ec8f5b8d33b7fc5d524c8fef79270cede4083f054f601d2ed"} Oct 31 13:02:54.696695 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:54.696650 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-28gmd" podStartSLOduration=6.238365963 podStartE2EDuration="16.696637557s" totalImagesPullingTime="10.458271594s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:02:54.696409787 +0000 UTC m=+17.759207173" watchObservedRunningTime="2026-10-31 13:02:54.696637557 +0000 UTC m=+17.759434941" Oct 31 13:02:55.075150 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:55.075116 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:55.075339 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:55.075242 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:02:55.075339 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:55.075288 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs podName:70a6eabd-c790-46fa-a9f1-cff156624c2f nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.075275341 +0000 UTC m=+34.138072717 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs") pod "network-metrics-daemon-sxmc2" (UID: "70a6eabd-c790-46fa-a9f1-cff156624c2f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:02:55.175445 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:55.175412 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54p8\" (UniqueName: \"kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8\") pod \"network-check-target-7f7h9\" (UID: \"018c5e83-597e-4c20-9063-5287b4c332a2\") " pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:55.175445 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:55.175463 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:55.175698 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:55.175575 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 13:02:55.175698 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:55.175610 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 13:02:55.175698 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:55.175633 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 13:02:55.175698 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:55.175643 2576 projected.go:196] Error preparing data for projected volume kube-api-access-n54p8 for pod openshift-network-diagnostics/network-check-target-7f7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:02:55.175698 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:55.175652 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret podName:2aee367b-5b87-4bfe-9620-5adf75e25b63 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.175635052 +0000 UTC m=+34.238432432 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret") pod "global-pull-secret-syncer-48bm7" (UID: "2aee367b-5b87-4bfe-9620-5adf75e25b63") : object "kube-system"/"original-pull-secret" not registered Oct 31 13:02:55.175698 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:55.175680 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8 podName:018c5e83-597e-4c20-9063-5287b4c332a2 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.175668744 +0000 UTC m=+34.238466144 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-n54p8" (UniqueName: "kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8") pod "network-check-target-7f7h9" (UID: "018c5e83-597e-4c20-9063-5287b4c332a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:02:55.449367 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:55.449291 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:55.449528 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:55.449406 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:02:56.449772 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:56.449737 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:56.449772 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:56.449774 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:56.450312 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:56.449864 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:02:56.450312 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:56.450007 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:02:57.449682 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:57.449646 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:57.449860 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:57.449780 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:02:58.449045 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:58.448868 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:02:58.449224 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:58.448902 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:02:58.449224 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:58.449111 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:02:58.449224 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:58.449213 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:02:58.679637 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:58.679532 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" event={"ID":"d2eab6ed-c191-4b42-899c-e816ab8d4a0b","Type":"ContainerStarted","Data":"cd3993ecc9785ce9cf7e43a3662c4d209cc21511ad8cd32f4ed7355b9ad22dc1"} Oct 31 13:02:58.679637 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:58.679629 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" event={"ID":"d2eab6ed-c191-4b42-899c-e816ab8d4a0b","Type":"ContainerStarted","Data":"be4dd456978e082e5f807339e7c9550889fcdcc030dc44d268f06f2cfec6ab7c"} Oct 31 13:02:58.680064 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:58.679672 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" event={"ID":"d2eab6ed-c191-4b42-899c-e816ab8d4a0b","Type":"ContainerStarted","Data":"3dd88e50eb992ec7173753f995f9ec8b95c7d62d2c75ecf142b941c7620d0fa8"} Oct 31 13:02:58.680064 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:58.679716 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" event={"ID":"d2eab6ed-c191-4b42-899c-e816ab8d4a0b","Type":"ContainerStarted","Data":"7cedef6cc65fec101fa51cbab895c023f5e2d6171bc6370416e869a049afb119"} Oct 31 13:02:58.680064 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:58.679762 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" event={"ID":"d2eab6ed-c191-4b42-899c-e816ab8d4a0b","Type":"ContainerStarted","Data":"36de682e46da3d0f7d620f746939b1637eb8053f3c627817b3587691fdba84d2"} Oct 31 13:02:58.680064 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:58.679812 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" event={"ID":"d2eab6ed-c191-4b42-899c-e816ab8d4a0b","Type":"ContainerStarted","Data":"9df0fc00a28d822b15b55e0923e6ebe52076a2e9e377c3abcf4168ac9d39aabe"} Oct 31 13:02:59.449751 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:02:59.449722 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:02:59.449915 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:02:59.449838 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:03:00.449366 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:00.449326 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:00.449731 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:00.449434 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:03:00.449731 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:00.449487 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:00.449731 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:00.449605 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:03:00.687003 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:00.686927 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" event={"ID":"d2eab6ed-c191-4b42-899c-e816ab8d4a0b","Type":"ContainerStarted","Data":"a49999eac56b1dd50195fbf9ea6e4228b236f73beeaf5caccb3dc6d77ce1738d"} Oct 31 13:03:01.449334 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:01.449300 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:01.449578 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:01.449399 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:03:02.449563 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:02.449535 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:02.449563 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:02.449565 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:02.450125 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:02.449668 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:03:02.450125 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:02.449791 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:03:02.694486 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:02.694414 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" event={"ID":"d2eab6ed-c191-4b42-899c-e816ab8d4a0b","Type":"ContainerStarted","Data":"405def12f446adb5fdf2f20c22b6a38173509a10cf93bd524fe25a77c98597a2"} Oct 31 13:03:02.694682 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:02.694666 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:02.710132 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:02.710105 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:02.772725 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:02.770537 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" podStartSLOduration=6.53761757 podStartE2EDuration="24.77052129s" totalImagesPullingTime="18.23290372s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:02.733861101 +0000 UTC m=+25.796658485" watchObservedRunningTime="2026-10-31 13:03:02.77052129 +0000 UTC m=+25.833318675" Oct 31 13:03:03.449065 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:03.449034 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:03.449211 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:03.449139 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:03:03.696672 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:03.696641 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:03.697170 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:03.696720 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:03.711742 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:03.711681 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:04.028115 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:04.027863 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sxmc2"] Oct 31 13:03:04.028253 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:04.028136 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:04.028253 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:04.028225 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:03:04.030829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:04.030803 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-48bm7"] Oct 31 13:03:04.030957 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:04.030905 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:04.031019 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:04.030997 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:03:04.031926 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:04.031459 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7f7h9"] Oct 31 13:03:04.031926 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:04.031911 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:04.032071 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:04.032024 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:03:05.448885 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:05.448848 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:05.449507 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:05.448860 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:05.449507 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:05.448938 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:03:05.449507 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:05.449019 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:03:05.449507 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:05.449054 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:05.449507 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:05.449107 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:03:07.449928 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:07.449896 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:07.450364 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:07.450002 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:03:07.450364 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:07.450009 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:07.450364 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:07.450035 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:07.450364 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:07.450093 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:03:07.450364 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:07.450139 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:03:09.449466 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:09.449433 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:09.449906 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:09.449447 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:09.449906 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:09.449574 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxmc2" podUID="70a6eabd-c790-46fa-a9f1-cff156624c2f" Oct 31 13:03:09.449906 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:09.449447 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:09.449906 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:09.449635 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-48bm7" podUID="2aee367b-5b87-4bfe-9620-5adf75e25b63" Oct 31 13:03:09.449906 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:09.449682 2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-7f7h9" podUID="018c5e83-597e-4c20-9063-5287b4c332a2" Oct 31 13:03:10.217107 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.217080 2576 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeReady" Oct 31 13:03:10.217264 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.217198 2576 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 31 13:03:10.292846 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.292812 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hgm5n"] Oct 31 13:03:10.327902 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.327874 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7rzwh"] Oct 31 13:03:10.328040 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.327999 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hgm5n" Oct 31 13:03:10.333254 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.333220 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 13:03:10.333397 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.333312 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 13:03:10.341606 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.341571 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hgm5n"] Oct 31 13:03:10.341694 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.341617 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7rzwh"] Oct 31 13:03:10.341694 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.341680 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:10.342152 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.342130 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 13:03:10.342257 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.342236 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wdnw7\"" Oct 31 13:03:10.433674 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.433642 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 13:03:10.443076 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.443054 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 13:03:10.443284 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.443272 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qbkfg\"" Oct 31 13:03:10.484776 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.484715 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sfsw\" (UniqueName: \"kubernetes.io/projected/e507f475-28b0-43c1-86b1-5ee3e4c4dad4-kube-api-access-4sfsw\") pod \"ingress-canary-hgm5n\" (UID: \"e507f475-28b0-43c1-86b1-5ee3e4c4dad4\") " pod="openshift-ingress-canary/ingress-canary-hgm5n" Oct 31 13:03:10.484776 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.484754 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0c07d8-f966-4acb-aea6-1ec8682b4877-config-volume\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:10.484776 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.484774 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:10.485120 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.484828 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bt8n\" (UniqueName: \"kubernetes.io/projected/ea0c07d8-f966-4acb-aea6-1ec8682b4877-kube-api-access-8bt8n\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:10.485120 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.484940 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea0c07d8-f966-4acb-aea6-1ec8682b4877-tmp-dir\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:10.485120 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.484958 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e507f475-28b0-43c1-86b1-5ee3e4c4dad4-cert\") pod \"ingress-canary-hgm5n\" (UID: \"e507f475-28b0-43c1-86b1-5ee3e4c4dad4\") " pod="openshift-ingress-canary/ingress-canary-hgm5n" Oct 31 13:03:10.586294 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.586260 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfsw\" (UniqueName: \"kubernetes.io/projected/e507f475-28b0-43c1-86b1-5ee3e4c4dad4-kube-api-access-4sfsw\") pod \"ingress-canary-hgm5n\" (UID: \"e507f475-28b0-43c1-86b1-5ee3e4c4dad4\") " pod="openshift-ingress-canary/ingress-canary-hgm5n" Oct 31 13:03:10.586294 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.586294 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0c07d8-f966-4acb-aea6-1ec8682b4877-config-volume\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:10.586487 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.586315 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:10.586487 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.586342 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bt8n\" (UniqueName: \"kubernetes.io/projected/ea0c07d8-f966-4acb-aea6-1ec8682b4877-kube-api-access-8bt8n\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:10.586487 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.586378 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea0c07d8-f966-4acb-aea6-1ec8682b4877-tmp-dir\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:10.586487 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.586400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e507f475-28b0-43c1-86b1-5ee3e4c4dad4-cert\") pod \"ingress-canary-hgm5n\" (UID: \"e507f475-28b0-43c1-86b1-5ee3e4c4dad4\") " pod="openshift-ingress-canary/ingress-canary-hgm5n" Oct 31 13:03:10.586487 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:10.586422 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 13:03:10.586487 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:10.586487 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls podName:ea0c07d8-f966-4acb-aea6-1ec8682b4877 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:11.086467299 +0000 UTC m=+34.149264680 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls") pod "dns-default-7rzwh" (UID: "ea0c07d8-f966-4acb-aea6-1ec8682b4877") : secret "dns-default-metrics-tls" not found Oct 31 13:03:10.586799 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.586755 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea0c07d8-f966-4acb-aea6-1ec8682b4877-tmp-dir\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:10.586923 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.586904 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0c07d8-f966-4acb-aea6-1ec8682b4877-config-volume\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:10.590092 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.590069 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e507f475-28b0-43c1-86b1-5ee3e4c4dad4-cert\") pod \"ingress-canary-hgm5n\" (UID: \"e507f475-28b0-43c1-86b1-5ee3e4c4dad4\") " pod="openshift-ingress-canary/ingress-canary-hgm5n" Oct 31 13:03:10.601402 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.601381 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bt8n\" (UniqueName: \"kubernetes.io/projected/ea0c07d8-f966-4acb-aea6-1ec8682b4877-kube-api-access-8bt8n\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:10.601625 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.601609 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfsw\" (UniqueName: \"kubernetes.io/projected/e507f475-28b0-43c1-86b1-5ee3e4c4dad4-kube-api-access-4sfsw\") pod \"ingress-canary-hgm5n\" (UID: \"e507f475-28b0-43c1-86b1-5ee3e4c4dad4\") " pod="openshift-ingress-canary/ingress-canary-hgm5n" Oct 31 13:03:10.650429 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.650405 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hgm5n" Oct 31 13:03:10.994460 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:10.993782 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hgm5n"] Oct 31 13:03:11.003532 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:11.003501 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode507f475_28b0_43c1_86b1_5ee3e4c4dad4.slice/crio-9d22eaca31a972fcd4c52990e3e01be055e6cbeedb68d4d20515d92c0ef46b79 WatchSource:0}: Error finding container 9d22eaca31a972fcd4c52990e3e01be055e6cbeedb68d4d20515d92c0ef46b79: Status 404 returned error can't find the container with id 9d22eaca31a972fcd4c52990e3e01be055e6cbeedb68d4d20515d92c0ef46b79 Oct 31 13:03:11.090768 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.090704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:11.090768 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.090753 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:11.090912 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:11.090842 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 13:03:11.090912 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:11.090849 2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:03:11.090912 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:11.090891 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls podName:ea0c07d8-f966-4acb-aea6-1ec8682b4877 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:12.090875871 +0000 UTC m=+35.153673234 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls") pod "dns-default-7rzwh" (UID: "ea0c07d8-f966-4acb-aea6-1ec8682b4877") : secret "dns-default-metrics-tls" not found Oct 31 13:03:11.090912 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:11.090902 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs podName:70a6eabd-c790-46fa-a9f1-cff156624c2f nodeName:}" failed. No retries permitted until 2026-10-31 13:03:43.090897005 +0000 UTC m=+66.153694367 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs") pod "network-metrics-daemon-sxmc2" (UID: "70a6eabd-c790-46fa-a9f1-cff156624c2f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 31 13:03:11.192061 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.192027 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54p8\" (UniqueName: \"kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8\") pod \"network-check-target-7f7h9\" (UID: \"018c5e83-597e-4c20-9063-5287b4c332a2\") " pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:11.192226 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.192076 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:11.192226 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:11.192191 2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 31 13:03:11.192331 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:11.192253 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret podName:2aee367b-5b87-4bfe-9620-5adf75e25b63 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:43.192235541 +0000 UTC m=+66.255032925 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret") pod "global-pull-secret-syncer-48bm7" (UID: "2aee367b-5b87-4bfe-9620-5adf75e25b63") : object "kube-system"/"original-pull-secret" not registered Oct 31 13:03:11.192331 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:11.192189 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 31 13:03:11.192331 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:11.192290 2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 31 13:03:11.192331 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:11.192303 2576 projected.go:196] Error preparing data for projected volume kube-api-access-n54p8 for pod openshift-network-diagnostics/network-check-target-7f7h9: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:03:11.192521 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:11.192350 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8 podName:018c5e83-597e-4c20-9063-5287b4c332a2 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:43.192336935 +0000 UTC m=+66.255134315 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-n54p8" (UniqueName: "kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8") pod "network-check-target-7f7h9" (UID: "018c5e83-597e-4c20-9063-5287b4c332a2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 31 13:03:11.452149 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.452114 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:11.452297 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.452176 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:11.452297 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.452197 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:11.457083 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.457060 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 13:03:11.457083 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.457071 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 13:03:11.458675 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.458645 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 13:03:11.458675 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.458659 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5rqjw\"" Oct 31 13:03:11.458846 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.458682 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wr5k7\"" Oct 31 13:03:11.458846 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.458706 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j8xvf\"" Oct 31 13:03:11.458846 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.458742 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 13:03:11.714651 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:11.714477 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hgm5n" event={"ID":"e507f475-28b0-43c1-86b1-5ee3e4c4dad4","Type":"ContainerStarted","Data":"9d22eaca31a972fcd4c52990e3e01be055e6cbeedb68d4d20515d92c0ef46b79"} Oct 31 13:03:12.100464 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:12.100389 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:12.103387 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:12.100791 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 13:03:12.103387 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:12.100892 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls podName:ea0c07d8-f966-4acb-aea6-1ec8682b4877 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:14.100858842 +0000 UTC m=+37.163656224 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls") pod "dns-default-7rzwh" (UID: "ea0c07d8-f966-4acb-aea6-1ec8682b4877") : secret "dns-default-metrics-tls" not found Oct 31 13:03:14.114944 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:14.114861 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:14.115454 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:14.115071 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 13:03:14.115454 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:14.115134 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls podName:ea0c07d8-f966-4acb-aea6-1ec8682b4877 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:18.115116439 +0000 UTC m=+41.177913819 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls") pod "dns-default-7rzwh" (UID: "ea0c07d8-f966-4acb-aea6-1ec8682b4877") : secret "dns-default-metrics-tls" not found Oct 31 13:03:15.725227 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:15.725194 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hgm5n" event={"ID":"e507f475-28b0-43c1-86b1-5ee3e4c4dad4","Type":"ContainerStarted","Data":"a4704e306936d7acaae644396bbf41080b0c81ecd071068d57ae14289fc0d14c"} Oct 31 13:03:15.744762 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:15.744695 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hgm5n" podStartSLOduration=1.890757277 podStartE2EDuration="5.744680071s" totalImagesPullingTime="3.853922794s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:15.744105404 +0000 UTC m=+38.806902789" watchObservedRunningTime="2026-10-31 13:03:15.744680071 +0000 UTC m=+38.807477457" Oct 31 13:03:17.158889 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.158853 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4"] Oct 31 13:03:17.180032 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.180007 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4"] Oct 31 13:03:17.180161 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.180095 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4" Oct 31 13:03:17.189735 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.189714 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 13:03:17.191207 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.191192 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-66zbj\"" Oct 31 13:03:17.196101 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.196085 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 13:03:17.233276 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.233250 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkj5f\" (UniqueName: \"kubernetes.io/projected/c913d3bd-b2b9-42d0-a02b-e0474c15e14c-kube-api-access-lkj5f\") pod \"migrator-66df5d496b-lnxq4\" (UID: \"c913d3bd-b2b9-42d0-a02b-e0474c15e14c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4" Oct 31 13:03:17.256404 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.256381 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-prddg-d22mn"] Oct 31 13:03:17.274702 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.274676 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4ftlz"] Oct 31 13:03:17.274877 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.274799 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.288274 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.288255 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-prddg-d22mn"] Oct 31 13:03:17.288380 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.288285 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-nlmx2"] Oct 31 13:03:17.288437 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.288379 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:17.293446 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.293429 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8dvth\"" Oct 31 13:03:17.294413 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.294394 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 13:03:17.294541 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.294470 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 13:03:17.294541 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.294472 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 13:03:17.299228 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.299212 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-b58t5\"" Oct 31 13:03:17.299341 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.299326 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 13:03:17.302439 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.302418 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4ftlz"] Oct 31 13:03:17.302524 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.302455 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nlmx2"] Oct 31 13:03:17.302562 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.302523 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.334231 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.334206 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-archives-path\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.334332 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.334274 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0b84e1d-390d-434d-b309-9440ec8195b6-data-volume\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.334332 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.334324 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.334433 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.334348 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f1b612e1-018b-47d0-880a-cb879202e312-signing-key\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:17.334469 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.334444 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfnsx\" (UniqueName: \"kubernetes.io/projected/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-kube-api-access-jfnsx\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.334557 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.334470 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q88wd\" (UniqueName: \"kubernetes.io/projected/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-api-access-q88wd\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.334557 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.334536 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f1b612e1-018b-47d0-880a-cb879202e312-signing-cabundle\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:17.334557 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.334550 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dk65\" (UniqueName: \"kubernetes.io/projected/f1b612e1-018b-47d0-880a-cb879202e312-kube-api-access-5dk65\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:17.334742 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.334575 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-trusted-ca-bundle\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.334742 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.334589 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.334742 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.334663 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkj5f\" (UniqueName: \"kubernetes.io/projected/c913d3bd-b2b9-42d0-a02b-e0474c15e14c-kube-api-access-lkj5f\") pod \"migrator-66df5d496b-lnxq4\" (UID: \"c913d3bd-b2b9-42d0-a02b-e0474c15e14c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4" Oct 31 13:03:17.334742 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.334682 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-service-ca-bundle\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.334742 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.334700 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0b84e1d-390d-434d-b309-9440ec8195b6-crio-socket\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.351696 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.351675 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkj5f\" (UniqueName: \"kubernetes.io/projected/c913d3bd-b2b9-42d0-a02b-e0474c15e14c-kube-api-access-lkj5f\") pod \"migrator-66df5d496b-lnxq4\" (UID: \"c913d3bd-b2b9-42d0-a02b-e0474c15e14c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4" Oct 31 13:03:17.391995 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.391971 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 13:03:17.392236 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.392222 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 13:03:17.392295 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.392254 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tmfz7\"" Oct 31 13:03:17.397952 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.397934 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 13:03:17.398123 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.398110 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 13:03:17.398174 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.398132 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 13:03:17.398279 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.398266 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 13:03:17.435379 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435333 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f1b612e1-018b-47d0-880a-cb879202e312-signing-cabundle\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:17.435379 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435357 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dk65\" (UniqueName: \"kubernetes.io/projected/f1b612e1-018b-47d0-880a-cb879202e312-kube-api-access-5dk65\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:17.435509 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435384 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-trusted-ca-bundle\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.435509 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435400 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.435509 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435419 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-service-ca-bundle\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.435509 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435441 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0b84e1d-390d-434d-b309-9440ec8195b6-crio-socket\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.435509 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435470 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-archives-path\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.435509 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:17.435494 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 13:03:17.435808 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:17.435567 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm podName:a0b84e1d-390d-434d-b309-9440ec8195b6 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:17.935547491 +0000 UTC m=+40.998344874 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nlmx2" (UID: "a0b84e1d-390d-434d-b309-9440ec8195b6") : configmap "kube-rbac-proxy" not found Oct 31 13:03:17.435808 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435614 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0b84e1d-390d-434d-b309-9440ec8195b6-data-volume\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.435808 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435655 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.435808 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435681 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f1b612e1-018b-47d0-880a-cb879202e312-signing-key\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:17.435808 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435700 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0b84e1d-390d-434d-b309-9440ec8195b6-crio-socket\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.435808 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435709 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfnsx\" (UniqueName: \"kubernetes.io/projected/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-kube-api-access-jfnsx\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.435808 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435732 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q88wd\" (UniqueName: \"kubernetes.io/projected/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-api-access-q88wd\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.436159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435927 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-archives-path\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.436159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.435990 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0b84e1d-390d-434d-b309-9440ec8195b6-data-volume\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.436159 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:17.436052 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 13:03:17.436159 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.436104 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-service-ca-bundle\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.436159 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:17.436108 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls podName:a0b84e1d-390d-434d-b309-9440ec8195b6 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:17.93609315 +0000 UTC m=+40.998890537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlmx2" (UID: "a0b84e1d-390d-434d-b309-9440ec8195b6") : secret "insights-runtime-extractor-tls" not found Oct 31 13:03:17.436383 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.436176 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f1b612e1-018b-47d0-880a-cb879202e312-signing-cabundle\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:17.436383 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.436316 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-trusted-ca-bundle\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.438241 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.438222 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f1b612e1-018b-47d0-880a-cb879202e312-signing-key\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:17.448161 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.448140 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dk65\" (UniqueName: \"kubernetes.io/projected/f1b612e1-018b-47d0-880a-cb879202e312-kube-api-access-5dk65\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:17.448306 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.448283 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfnsx\" (UniqueName: \"kubernetes.io/projected/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-kube-api-access-jfnsx\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.448383 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.448360 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q88wd\" (UniqueName: \"kubernetes.io/projected/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-api-access-q88wd\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.505150 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.505129 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4" Oct 31 13:03:17.596752 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.596557 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:17.614883 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.614861 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:17.814448 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.814248 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4"] Oct 31 13:03:17.817434 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:17.817407 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc913d3bd_b2b9_42d0_a02b_e0474c15e14c.slice/crio-d931074a761f4f450c2030c9edca2919d06d28e256e68d606a39bd28608a996e WatchSource:0}: Error finding container d931074a761f4f450c2030c9edca2919d06d28e256e68d606a39bd28608a996e: Status 404 returned error can't find the container with id d931074a761f4f450c2030c9edca2919d06d28e256e68d606a39bd28608a996e Oct 31 13:03:17.885885 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.885853 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-prddg-d22mn"] Oct 31 13:03:17.888872 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:17.888844 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf935cb_694a_4d54_b7a3_f0be4bc1212c.slice/crio-1dcd59a48b4c83fd45e6638c9df05b23c118bbfdcc28b472a48f59b8bf4e3f90 WatchSource:0}: Error finding container 1dcd59a48b4c83fd45e6638c9df05b23c118bbfdcc28b472a48f59b8bf4e3f90: Status 404 returned error can't find the container with id 1dcd59a48b4c83fd45e6638c9df05b23c118bbfdcc28b472a48f59b8bf4e3f90 Oct 31 13:03:17.924615 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.924576 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-4ftlz"] Oct 31 13:03:17.927484 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:17.927464 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b612e1_018b_47d0_880a_cb879202e312.slice/crio-e4ee2b25d23199c272ed6f880b4618f67728b41aaa3a25d3ae9c230aa91e9af1 WatchSource:0}: Error finding container e4ee2b25d23199c272ed6f880b4618f67728b41aaa3a25d3ae9c230aa91e9af1: Status 404 returned error can't find the container with id e4ee2b25d23199c272ed6f880b4618f67728b41aaa3a25d3ae9c230aa91e9af1 Oct 31 13:03:17.938939 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.938919 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.939011 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:17.938988 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:17.939046 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:17.939027 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 13:03:17.939091 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:17.939081 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm podName:a0b84e1d-390d-434d-b309-9440ec8195b6 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:18.939066065 +0000 UTC m=+42.001863433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nlmx2" (UID: "a0b84e1d-390d-434d-b309-9440ec8195b6") : configmap "kube-rbac-proxy" not found Oct 31 13:03:17.939091 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:17.939085 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 13:03:17.939156 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:17.939120 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls podName:a0b84e1d-390d-434d-b309-9440ec8195b6 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:18.939109472 +0000 UTC m=+42.001906834 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlmx2" (UID: "a0b84e1d-390d-434d-b309-9440ec8195b6") : secret "insights-runtime-extractor-tls" not found Oct 31 13:03:18.140865 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:18.140792 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:18.141020 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:18.140934 2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 31 13:03:18.141020 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:18.140997 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls podName:ea0c07d8-f966-4acb-aea6-1ec8682b4877 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:26.14098055 +0000 UTC m=+49.203777913 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls") pod "dns-default-7rzwh" (UID: "ea0c07d8-f966-4acb-aea6-1ec8682b4877") : secret "dns-default-metrics-tls" not found Oct 31 13:03:18.735161 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:18.735122 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" event={"ID":"f1b612e1-018b-47d0-880a-cb879202e312","Type":"ContainerStarted","Data":"e4ee2b25d23199c272ed6f880b4618f67728b41aaa3a25d3ae9c230aa91e9af1"} Oct 31 13:03:18.736648 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:18.736621 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-prddg-d22mn" event={"ID":"ccf935cb-694a-4d54-b7a3-f0be4bc1212c","Type":"ContainerStarted","Data":"1dcd59a48b4c83fd45e6638c9df05b23c118bbfdcc28b472a48f59b8bf4e3f90"} Oct 31 13:03:18.738416 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:18.738392 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4" event={"ID":"c913d3bd-b2b9-42d0-a02b-e0474c15e14c","Type":"ContainerStarted","Data":"d931074a761f4f450c2030c9edca2919d06d28e256e68d606a39bd28608a996e"} Oct 31 13:03:18.947607 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:18.947563 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:18.947790 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:18.947669 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:18.947790 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:18.947752 2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 31 13:03:18.947864 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:18.947813 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm podName:a0b84e1d-390d-434d-b309-9440ec8195b6 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:20.947794001 +0000 UTC m=+44.010591386 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm") pod "insights-runtime-extractor-nlmx2" (UID: "a0b84e1d-390d-434d-b309-9440ec8195b6") : configmap "kube-rbac-proxy" not found Oct 31 13:03:18.947864 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:18.947750 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 13:03:18.947931 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:18.947891 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls podName:a0b84e1d-390d-434d-b309-9440ec8195b6 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:20.947871592 +0000 UTC m=+44.010668961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlmx2" (UID: "a0b84e1d-390d-434d-b309-9440ec8195b6") : secret "insights-runtime-extractor-tls" not found Oct 31 13:03:20.747662 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:20.747419 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4" event={"ID":"c913d3bd-b2b9-42d0-a02b-e0474c15e14c","Type":"ContainerStarted","Data":"8a665b25de4924dc4f00ec721df2b8358752440f0efb162e8ed48c9765364e40"} Oct 31 13:03:20.767544 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:20.766549 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4" podStartSLOduration=1.178867323 podStartE2EDuration="3.766530372s" totalImagesPullingTime="2.587663049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:20.766169132 +0000 UTC m=+43.828966544" watchObservedRunningTime="2026-10-31 13:03:20.766530372 +0000 UTC m=+43.829327758" Oct 31 13:03:20.964533 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:20.964496 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:20.964744 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:20.964623 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:20.964821 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:20.964783 2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 31 13:03:20.964876 ip-10-0-128-41 kubenswrapper[2576]: E1031 13:03:20.964849 2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls podName:a0b84e1d-390d-434d-b309-9440ec8195b6 nodeName:}" failed. No retries permitted until 2026-10-31 13:03:24.964832819 +0000 UTC m=+48.027630187 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls") pod "insights-runtime-extractor-nlmx2" (UID: "a0b84e1d-390d-434d-b309-9440ec8195b6") : secret "insights-runtime-extractor-tls" not found Oct 31 13:03:20.965187 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:20.965157 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:21.751834 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:21.751583 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-prddg-d22mn" event={"ID":"ccf935cb-694a-4d54-b7a3-f0be4bc1212c","Type":"ContainerStarted","Data":"7972a59d25d88bce295d3bafbaa2543562203f9c595d50562c67dac3744b6338"} Oct 31 13:03:21.753047 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:21.753016 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" event={"ID":"f1b612e1-018b-47d0-880a-cb879202e312","Type":"ContainerStarted","Data":"0391633f48fb1b5c6ac8d934fa39eece91cd8612e280db795be9084642691a6d"} Oct 31 13:03:21.771627 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:21.771564 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-prddg-d22mn" podStartSLOduration=1.2917786900000001 podStartE2EDuration="4.771548669s" totalImagesPullingTime="3.479769979s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:21.770263663 +0000 UTC m=+44.833061049" watchObservedRunningTime="2026-10-31 13:03:21.771548669 +0000 UTC m=+44.834346055" Oct 31 13:03:21.788449 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:21.788409 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" podStartSLOduration=1.338617115 podStartE2EDuration="4.788396351s" totalImagesPullingTime="3.449779236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:21.78812415 +0000 UTC m=+44.850921536" watchObservedRunningTime="2026-10-31 13:03:21.788396351 +0000 UTC m=+44.851193749" Oct 31 13:03:24.415392 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:24.415361 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l5vvf_dc1ff283-237c-45f3-a90f-88cb75bcb863/dns-node-resolver/0.log" Oct 31 13:03:24.998244 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:24.998205 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:25.002246 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:25.002210 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:25.169396 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:25.169363 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:25.216580 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:25.216399 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fbstt_54a25982-6507-405a-8e10-81d965ab7f4f/node-ca/0.log" Oct 31 13:03:25.443019 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:25.442991 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-nlmx2"] Oct 31 13:03:25.446006 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:25.445981 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0b84e1d_390d_434d_b309_9440ec8195b6.slice/crio-27bdeb9c4bd1d0a81f9456ba6c9c6332548e508993091dbfeffce7c648b3db76 WatchSource:0}: Error finding container 27bdeb9c4bd1d0a81f9456ba6c9c6332548e508993091dbfeffce7c648b3db76: Status 404 returned error can't find the container with id 27bdeb9c4bd1d0a81f9456ba6c9c6332548e508993091dbfeffce7c648b3db76 Oct 31 13:03:25.763662 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:25.763579 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nlmx2" event={"ID":"a0b84e1d-390d-434d-b309-9440ec8195b6","Type":"ContainerStarted","Data":"05c5ca511ad8f8295f105ea2ed59b8123c59cab2f02357f0edc8f861e1b9ce15"} Oct 31 13:03:25.763662 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:25.763645 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nlmx2" event={"ID":"a0b84e1d-390d-434d-b309-9440ec8195b6","Type":"ContainerStarted","Data":"27bdeb9c4bd1d0a81f9456ba6c9c6332548e508993091dbfeffce7c648b3db76"} Oct 31 13:03:26.016168 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:26.016098 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hgm5n_e507f475-28b0-43c1-86b1-5ee3e4c4dad4/serve-healthcheck-canary/0.log" Oct 31 13:03:26.206321 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:26.206296 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:26.208577 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:26.208555 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:26.279477 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:26.279408 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:26.553152 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:26.553085 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7rzwh"] Oct 31 13:03:26.555633 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:26.555607 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea0c07d8_f966_4acb_aea6_1ec8682b4877.slice/crio-9e2e00a5cedbfc0ca16983e327195326fb3f7628e80e8fb45e7f79c479c139cf WatchSource:0}: Error finding container 9e2e00a5cedbfc0ca16983e327195326fb3f7628e80e8fb45e7f79c479c139cf: Status 404 returned error can't find the container with id 9e2e00a5cedbfc0ca16983e327195326fb3f7628e80e8fb45e7f79c479c139cf Oct 31 13:03:26.617325 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:26.617303 2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-lnxq4_c913d3bd-b2b9-42d0-a02b-e0474c15e14c/migrator/0.log" Oct 31 13:03:26.768565 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:26.768516 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nlmx2" event={"ID":"a0b84e1d-390d-434d-b309-9440ec8195b6","Type":"ContainerStarted","Data":"69abb6243325eb31771b2a983ad039f054700013939fd642166bcd21fea8360b"} Oct 31 13:03:26.769798 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:26.769768 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rzwh" event={"ID":"ea0c07d8-f966-4acb-aea6-1ec8682b4877","Type":"ContainerStarted","Data":"9e2e00a5cedbfc0ca16983e327195326fb3f7628e80e8fb45e7f79c479c139cf"} Oct 31 13:03:28.778139 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:28.778032 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-nlmx2" event={"ID":"a0b84e1d-390d-434d-b309-9440ec8195b6","Type":"ContainerStarted","Data":"8f481092da58c4a7ebae039a833fb68549ab9e0fa71c2cbcad4e29b6a3a9d14e"} Oct 31 13:03:28.779923 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:28.779897 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rzwh" event={"ID":"ea0c07d8-f966-4acb-aea6-1ec8682b4877","Type":"ContainerStarted","Data":"b2f03e60a05d2f53e2ff509624ff8d797ac41b1cc0c7ad6ff232e39505043c0a"} Oct 31 13:03:28.780037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:28.779942 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7rzwh" event={"ID":"ea0c07d8-f966-4acb-aea6-1ec8682b4877","Type":"ContainerStarted","Data":"eff7d57cc576c47f6d84197dcaf2e41c7e7e70edb343502795e812adb0385c4e"} Oct 31 13:03:28.780037 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:28.779963 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:28.801364 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:28.801313 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-nlmx2" podStartSLOduration=9.44919446 podStartE2EDuration="11.801300296s" totalImagesPullingTime="2.352105836s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 13:03:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:28.799748608 +0000 UTC m=+51.862546023" watchObservedRunningTime="2026-10-31 13:03:28.801300296 +0000 UTC m=+51.864097681" Oct 31 13:03:28.821302 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:28.821252 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-7rzwh" podStartSLOduration=16.943126469 podStartE2EDuration="18.821238145s" totalImagesPullingTime="1.878111676s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:28.820795379 +0000 UTC m=+51.883592786" watchObservedRunningTime="2026-10-31 13:03:28.821238145 +0000 UTC m=+51.884035530" Oct 31 13:03:35.710667 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:35.710639 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:38.785718 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:38.785689 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:40.375029 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.374995 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-xc5m4"] Oct 31 13:03:40.379418 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.379398 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:03:40.381314 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.381289 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj"] Oct 31 13:03:40.384273 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.384252 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" Oct 31 13:03:40.388861 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.388832 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 13:03:40.388960 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.388896 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 13:03:40.388960 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.388917 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-zdl2b\"" Oct 31 13:03:40.397915 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.397894 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj"] Oct 31 13:03:40.401114 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.401092 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-xc5m4"] Oct 31 13:03:40.488991 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.488959 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 31 13:03:40.488991 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.488959 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lmr7q\"" Oct 31 13:03:40.511639 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.511609 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70ea50fe-3024-4a82-84a5-5c236706402b-tmp\") pod \"downloads-6dd789c8b7-xc5m4\" (UID: \"70ea50fe-3024-4a82-84a5-5c236706402b\") " pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:03:40.511773 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.511675 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qj4l\" (UniqueName: \"kubernetes.io/projected/70ea50fe-3024-4a82-84a5-5c236706402b-kube-api-access-7qj4l\") pod \"downloads-6dd789c8b7-xc5m4\" (UID: \"70ea50fe-3024-4a82-84a5-5c236706402b\") " pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:03:40.511811 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.511773 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25233e95-abc9-4f04-b7b2-65933229d471-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5vhhj\" (UID: \"25233e95-abc9-4f04-b7b2-65933229d471\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" Oct 31 13:03:40.612532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.612493 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25233e95-abc9-4f04-b7b2-65933229d471-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5vhhj\" (UID: \"25233e95-abc9-4f04-b7b2-65933229d471\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" Oct 31 13:03:40.612724 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.612539 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70ea50fe-3024-4a82-84a5-5c236706402b-tmp\") pod \"downloads-6dd789c8b7-xc5m4\" (UID: \"70ea50fe-3024-4a82-84a5-5c236706402b\") " pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:03:40.612724 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.612638 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qj4l\" (UniqueName: \"kubernetes.io/projected/70ea50fe-3024-4a82-84a5-5c236706402b-kube-api-access-7qj4l\") pod \"downloads-6dd789c8b7-xc5m4\" (UID: \"70ea50fe-3024-4a82-84a5-5c236706402b\") " pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:03:40.612947 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.612927 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70ea50fe-3024-4a82-84a5-5c236706402b-tmp\") pod \"downloads-6dd789c8b7-xc5m4\" (UID: \"70ea50fe-3024-4a82-84a5-5c236706402b\") " pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:03:40.615136 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.615111 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25233e95-abc9-4f04-b7b2-65933229d471-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5vhhj\" (UID: \"25233e95-abc9-4f04-b7b2-65933229d471\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" Oct 31 13:03:40.626210 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.626159 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qj4l\" (UniqueName: \"kubernetes.io/projected/70ea50fe-3024-4a82-84a5-5c236706402b-kube-api-access-7qj4l\") pod \"downloads-6dd789c8b7-xc5m4\" (UID: \"70ea50fe-3024-4a82-84a5-5c236706402b\") " pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:03:40.692852 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.692825 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:03:40.697829 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.697811 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" Oct 31 13:03:40.973000 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.972971 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj"] Oct 31 13:03:40.975519 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:40.975498 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-xc5m4"] Oct 31 13:03:40.976270 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:40.976245 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25233e95_abc9_4f04_b7b2_65933229d471.slice/crio-28b3c56c45fc5879b2fd1ec3d1cd8238d569fbebce0d4ff34da1b76e2d06b84e WatchSource:0}: Error finding container 28b3c56c45fc5879b2fd1ec3d1cd8238d569fbebce0d4ff34da1b76e2d06b84e: Status 404 returned error can't find the container with id 28b3c56c45fc5879b2fd1ec3d1cd8238d569fbebce0d4ff34da1b76e2d06b84e Oct 31 13:03:40.979226 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:40.979205 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ea50fe_3024_4a82_84a5_5c236706402b.slice/crio-3045860abe8b538dad79b8b05d09704f6971fba3add3cf2465b74ce1928ba9cc WatchSource:0}: Error finding container 3045860abe8b538dad79b8b05d09704f6971fba3add3cf2465b74ce1928ba9cc: Status 404 returned error can't find the container with id 3045860abe8b538dad79b8b05d09704f6971fba3add3cf2465b74ce1928ba9cc Oct 31 13:03:41.825054 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:41.824983 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" event={"ID":"25233e95-abc9-4f04-b7b2-65933229d471","Type":"ContainerStarted","Data":"28b3c56c45fc5879b2fd1ec3d1cd8238d569fbebce0d4ff34da1b76e2d06b84e"} Oct 31 13:03:41.826512 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:41.826479 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-xc5m4" event={"ID":"70ea50fe-3024-4a82-84a5-5c236706402b","Type":"ContainerStarted","Data":"3045860abe8b538dad79b8b05d09704f6971fba3add3cf2465b74ce1928ba9cc"} Oct 31 13:03:42.834364 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:42.834327 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" event={"ID":"25233e95-abc9-4f04-b7b2-65933229d471","Type":"ContainerStarted","Data":"56567784a0b8db9485b5f66de2998c7dbbb9776ac6fbd5706243848a2bb42855"} Oct 31 13:03:42.834969 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:42.834558 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" Oct 31 13:03:42.840258 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:42.840231 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" Oct 31 13:03:42.854949 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:42.854899 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" podStartSLOduration=1.6150929280000001 podStartE2EDuration="2.854884944s" totalImagesPullingTime="1.239792016s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:42.853714048 +0000 UTC m=+65.916511465" watchObservedRunningTime="2026-10-31 13:03:42.854884944 +0000 UTC m=+65.917682330" Oct 31 13:03:43.133589 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.133507 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:43.136454 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.136433 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 13:03:43.146722 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.146687 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:43.234546 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.234510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54p8\" (UniqueName: \"kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8\") pod \"network-check-target-7f7h9\" (UID: \"018c5e83-597e-4c20-9063-5287b4c332a2\") " pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:43.234546 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.234546 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:43.237639 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.237617 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 13:03:43.237706 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.237693 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 13:03:43.247414 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.247387 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:43.247533 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.247513 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 13:03:43.258055 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.258031 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n54p8\" (UniqueName: \"kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8\") pod \"network-check-target-7f7h9\" (UID: \"018c5e83-597e-4c20-9063-5287b4c332a2\") " pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:43.310395 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.310365 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j8xvf\"" Oct 31 13:03:43.318431 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.318407 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:43.324206 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.324186 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5rqjw\"" Oct 31 13:03:43.331289 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.331269 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:43.350301 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.350277 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wr5k7\"" Oct 31 13:03:43.358937 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.358579 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:43.594314 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.594277 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-7f7h9"] Oct 31 13:03:43.597173 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:43.597145 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod018c5e83_597e_4c20_9063_5287b4c332a2.slice/crio-c75cc3474d609218febf8d3a1929a443940db8c99210ddf4b15a4bc4bfab4ca5 WatchSource:0}: Error finding container c75cc3474d609218febf8d3a1929a443940db8c99210ddf4b15a4bc4bfab4ca5: Status 404 returned error can't find the container with id c75cc3474d609218febf8d3a1929a443940db8c99210ddf4b15a4bc4bfab4ca5 Oct 31 13:03:43.618992 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.618965 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-48bm7"] Oct 31 13:03:43.622071 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:43.622039 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aee367b_5b87_4bfe_9620_5adf75e25b63.slice/crio-365e4ac477b108f1e883c86733c6bf1d41c727c74d0fa2e18b97e0ec2abcbf90 WatchSource:0}: Error finding container 365e4ac477b108f1e883c86733c6bf1d41c727c74d0fa2e18b97e0ec2abcbf90: Status 404 returned error can't find the container with id 365e4ac477b108f1e883c86733c6bf1d41c727c74d0fa2e18b97e0ec2abcbf90 Oct 31 13:03:43.658376 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.658352 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sxmc2"] Oct 31 13:03:43.661665 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:43.661633 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a6eabd_c790_46fa_a9f1_cff156624c2f.slice/crio-7113b20c24c9ecc50668af315409491a04bf884757624db5c480ebdf7e55509c WatchSource:0}: Error finding container 7113b20c24c9ecc50668af315409491a04bf884757624db5c480ebdf7e55509c: Status 404 returned error can't find the container with id 7113b20c24c9ecc50668af315409491a04bf884757624db5c480ebdf7e55509c Oct 31 13:03:43.844880 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.844786 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7f7h9" event={"ID":"018c5e83-597e-4c20-9063-5287b4c332a2","Type":"ContainerStarted","Data":"c75cc3474d609218febf8d3a1929a443940db8c99210ddf4b15a4bc4bfab4ca5"} Oct 31 13:03:43.846114 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.846083 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-48bm7" event={"ID":"2aee367b-5b87-4bfe-9620-5adf75e25b63","Type":"ContainerStarted","Data":"365e4ac477b108f1e883c86733c6bf1d41c727c74d0fa2e18b97e0ec2abcbf90"} Oct 31 13:03:43.847384 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.847353 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sxmc2" event={"ID":"70a6eabd-c790-46fa-a9f1-cff156624c2f","Type":"ContainerStarted","Data":"7113b20c24c9ecc50668af315409491a04bf884757624db5c480ebdf7e55509c"} Oct 31 13:03:43.927053 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.926980 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-64885574d7-l7vzc"] Oct 31 13:03:43.931074 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.931058 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:43.936008 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.935982 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 13:03:43.936131 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.936050 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 13:03:43.936185 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.936175 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mk4zx\"" Oct 31 13:03:43.936227 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.936185 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 13:03:43.936280 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.936259 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 13:03:43.936527 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.936508 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 13:03:43.941767 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:43.941727 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64885574d7-l7vzc"] Oct 31 13:03:44.043356 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.043314 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-config\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.043526 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.043379 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d38ca15-69a2-4950-9bf5-c77868a86cfb-tmp\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.043526 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.043471 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-oauth-config\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.043526 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.043504 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-serving-cert\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.043526 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.043521 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ml8wx\" (UniqueName: \"kubernetes.io/projected/4d38ca15-69a2-4950-9bf5-c77868a86cfb-kube-api-access-ml8wx\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.043773 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.043537 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-oauth-serving-cert\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.043773 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.043563 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-service-ca\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.144727 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.144674 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-serving-cert\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.144727 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.144724 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8wx\" (UniqueName: \"kubernetes.io/projected/4d38ca15-69a2-4950-9bf5-c77868a86cfb-kube-api-access-ml8wx\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.145094 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.145056 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-oauth-serving-cert\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.145250 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.145096 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-service-ca\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.145250 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.145134 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-config\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.145250 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.145224 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d38ca15-69a2-4950-9bf5-c77868a86cfb-tmp\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.145417 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.145263 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-oauth-config\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.145890 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.145862 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d38ca15-69a2-4950-9bf5-c77868a86cfb-tmp\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.146007 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.145954 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-oauth-serving-cert\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.146007 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.145952 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-service-ca\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.146132 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.146072 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-config\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.147616 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.147570 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-serving-cert\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.147842 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.147817 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-oauth-config\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.158532 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.158498 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8wx\" (UniqueName: \"kubernetes.io/projected/4d38ca15-69a2-4950-9bf5-c77868a86cfb-kube-api-access-ml8wx\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.245846 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.245519 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:44.535910 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.535813 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64885574d7-l7vzc"] Oct 31 13:03:44.853847 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:44.853722 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64885574d7-l7vzc" event={"ID":"4d38ca15-69a2-4950-9bf5-c77868a86cfb","Type":"ContainerStarted","Data":"0db026a30d618fbaa7375a75aa984d06041c8e566648bb59d6e58e390eee588b"} Oct 31 13:03:45.860009 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:45.859436 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sxmc2" event={"ID":"70a6eabd-c790-46fa-a9f1-cff156624c2f","Type":"ContainerStarted","Data":"3b3c667f292ce3fba18d6456a4b87f42fe8510ce9d57afa85afb6f67ea81cf30"} Oct 31 13:03:45.860009 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:45.859497 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sxmc2" event={"ID":"70a6eabd-c790-46fa-a9f1-cff156624c2f","Type":"ContainerStarted","Data":"1e489c4a81d0c0bc0aa1696bd3489be5fb4083f3db71b0a9e5a70a958261f281"} Oct 31 13:03:45.883259 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:45.881860 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sxmc2" podStartSLOduration=66.392368555 podStartE2EDuration="1m7.881841977s" totalImagesPullingTime="1.489473422s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:45.881577803 +0000 UTC m=+68.944375190" watchObservedRunningTime="2026-10-31 13:03:45.881841977 +0000 UTC m=+68.944639362" Oct 31 13:03:47.483674 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.483635 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-949gv"] Oct 31 13:03:47.487517 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.487490 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.495675 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.495651 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 13:03:47.498033 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.498013 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 13:03:47.498168 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.498152 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 13:03:47.501212 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.501166 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-p4sq4\"" Oct 31 13:03:47.501310 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.501273 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 13:03:47.501666 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.501633 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 13:03:47.502920 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.502878 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-949gv"] Oct 31 13:03:47.509182 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.509162 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p5fj4"] Oct 31 13:03:47.513346 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.513321 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.530420 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.530375 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rrldl\"" Oct 31 13:03:47.530547 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.530484 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 13:03:47.530547 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.530524 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 13:03:47.573361 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.572744 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-sys\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.573361 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.572790 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-tls\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.573361 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.572823 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.573361 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.572857 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.573361 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.572886 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-wtmp\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.573361 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.572914 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-textfile\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.573361 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.572943 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.573361 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.572969 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/896e7579-dea0-49ee-92c5-cddf258c6905-metrics-client-ca\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.573361 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.573004 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.573361 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.573058 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.573361 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.573089 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxr4p\" (UniqueName: \"kubernetes.io/projected/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-kube-api-access-vxr4p\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.573361 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.573112 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-root\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.573361 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.573137 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm6fc\" (UniqueName: \"kubernetes.io/projected/896e7579-dea0-49ee-92c5-cddf258c6905-kube-api-access-tm6fc\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.595461 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.595433 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 13:03:47.673700 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.673429 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-sys\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.673700 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.673476 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-tls\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.673700 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.673510 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.673700 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.673545 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.673700 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.673576 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-wtmp\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.673700 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.673545 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-sys\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.675162 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.675136 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-wtmp\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.675300 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.675168 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-textfile\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.675300 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.675192 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.675300 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.675213 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/896e7579-dea0-49ee-92c5-cddf258c6905-metrics-client-ca\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.675300 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.675249 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.675514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.675306 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.675514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.675346 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxr4p\" (UniqueName: \"kubernetes.io/projected/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-kube-api-access-vxr4p\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.675514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.675371 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-root\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.675514 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.675397 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6fc\" (UniqueName: \"kubernetes.io/projected/896e7579-dea0-49ee-92c5-cddf258c6905-kube-api-access-tm6fc\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.676043 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.676015 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-textfile\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.678056 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.676786 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.678056 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.677234 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/896e7579-dea0-49ee-92c5-cddf258c6905-metrics-client-ca\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.678056 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.677785 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.678056 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.677922 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.678056 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.677997 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-root\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.679672 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.679646 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-tls\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.680943 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.680888 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.689289 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.689266 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.689544 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.689521 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm6fc\" (UniqueName: \"kubernetes.io/projected/896e7579-dea0-49ee-92c5-cddf258c6905-kube-api-access-tm6fc\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.694079 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.694061 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxr4p\" (UniqueName: \"kubernetes.io/projected/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-kube-api-access-vxr4p\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.811059 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.810963 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:47.835230 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.835030 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:47.863380 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:47.863334 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod896e7579_dea0_49ee_92c5_cddf258c6905.slice/crio-5f93fe1ef3eec4e3c5b2f435d20d4de70c17d1dab9b05cb13fcec39e05d94004 WatchSource:0}: Error finding container 5f93fe1ef3eec4e3c5b2f435d20d4de70c17d1dab9b05cb13fcec39e05d94004: Status 404 returned error can't find the container with id 5f93fe1ef3eec4e3c5b2f435d20d4de70c17d1dab9b05cb13fcec39e05d94004 Oct 31 13:03:47.897951 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:47.897774 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5fj4" event={"ID":"896e7579-dea0-49ee-92c5-cddf258c6905","Type":"ContainerStarted","Data":"5f93fe1ef3eec4e3c5b2f435d20d4de70c17d1dab9b05cb13fcec39e05d94004"} Oct 31 13:03:48.112614 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.112562 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-949gv"] Oct 31 13:03:48.115995 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:48.115941 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e3f5e50_ca1a_4a2a_af47_30a240b36bd5.slice/crio-61946c5f5c049e16bfb07ca0280dc21252438eb04d18b2e396b56e81454467f8 WatchSource:0}: Error finding container 61946c5f5c049e16bfb07ca0280dc21252438eb04d18b2e396b56e81454467f8: Status 404 returned error can't find the container with id 61946c5f5c049e16bfb07ca0280dc21252438eb04d18b2e396b56e81454467f8 Oct 31 13:03:48.575621 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.574613 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 13:03:48.582469 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.582433 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.587887 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.587860 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 13:03:48.588257 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.588062 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nztnk\"" Oct 31 13:03:48.588257 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.588155 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 13:03:48.588257 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.588165 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 13:03:48.588257 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.588222 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 13:03:48.588515 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.588502 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 13:03:48.589819 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.589655 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 13:03:48.589819 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.589706 2576 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 13:03:48.589819 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.589708 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 13:03:48.589819 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.589769 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 13:03:48.599459 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.599433 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 13:03:48.683819 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.683781 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-out\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.683989 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.683827 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.683989 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.683852 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.683989 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.683885 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdn2q\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-kube-api-access-xdn2q\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.683989 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.683914 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.683989 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.683942 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.683989 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.683974 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.684294 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.683997 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.684294 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.684016 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.684294 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.684034 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-web-config\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.684294 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.684050 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.684294 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.684072 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.684294 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.684100 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.784817 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.784776 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdn2q\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-kube-api-access-xdn2q\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.784817 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.784820 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.785038 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.784844 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.785038 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.784860 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.785038 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.784877 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.785038 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.784896 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.785038 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.784916 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-web-config\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.785038 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.784943 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.785038 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.784963 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.785038 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.784982 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.785038 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.785005 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-out\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.785038 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.785024 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.785038 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.785046 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.786730 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.786696 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.787005 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.786978 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.788582 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.788542 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-out\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.788680 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.788618 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.790161 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.790113 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.790414 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.790396 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.791316 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.791290 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.791764 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.791739 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-web-config\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.791908 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.791851 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.792186 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.792163 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.792550 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.792532 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.792893 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.792872 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.805737 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.805689 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdn2q\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-kube-api-access-xdn2q\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:48.910555 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.910405 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" event={"ID":"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5","Type":"ContainerStarted","Data":"f11e741d0ee98aa89db01464d2c41e5a3c4f4f7a71abe9e28d5530fd4404b6d9"} Oct 31 13:03:48.910555 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.910479 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" event={"ID":"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5","Type":"ContainerStarted","Data":"d06d42b09175eb79bbb3102352f332cb175a761dd56017319ed38d0dabdf35cb"} Oct 31 13:03:48.910555 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.910508 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" event={"ID":"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5","Type":"ContainerStarted","Data":"61946c5f5c049e16bfb07ca0280dc21252438eb04d18b2e396b56e81454467f8"} Oct 31 13:03:48.916299 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:48.916050 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:49.243735 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.243696 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 13:03:49.573516 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.573438 2576 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57c987d54b-lc452"] Oct 31 13:03:49.577628 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.577581 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.591225 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.591199 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 13:03:49.591342 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.591254 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 13:03:49.591342 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.591304 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 13:03:49.591691 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.591567 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 13:03:49.591691 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.591573 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-25mffia9fn4bj\"" Oct 31 13:03:49.591691 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.591684 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qx7z5\"" Oct 31 13:03:49.591932 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.591701 2576 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 13:03:49.593215 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.593117 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57c987d54b-lc452"] Oct 31 13:03:49.692933 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.692832 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.692933 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.692893 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.693155 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.692965 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-grpc-tls\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.693155 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.693083 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7mts\" (UniqueName: \"kubernetes.io/projected/e25856e6-e50d-48d2-94a5-8966be78dca4-kube-api-access-k7mts\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.693155 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.693114 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.693155 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.693150 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.693357 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.693178 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e25856e6-e50d-48d2-94a5-8966be78dca4-metrics-client-ca\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.693357 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.693202 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-tls\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.693357 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.693224 2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e25856e6-e50d-48d2-94a5-8966be78dca4-tmp\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.794683 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.794647 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mts\" (UniqueName: \"kubernetes.io/projected/e25856e6-e50d-48d2-94a5-8966be78dca4-kube-api-access-k7mts\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.794808 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.794704 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.794808 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.794745 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.794808 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.794790 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e25856e6-e50d-48d2-94a5-8966be78dca4-metrics-client-ca\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.794972 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.794816 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-tls\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.794972 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.794839 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e25856e6-e50d-48d2-94a5-8966be78dca4-tmp\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.794972 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.794868 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.794972 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.794908 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.794972 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.794931 2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-grpc-tls\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.795398 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.795352 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e25856e6-e50d-48d2-94a5-8966be78dca4-tmp\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.795784 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.795758 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e25856e6-e50d-48d2-94a5-8966be78dca4-metrics-client-ca\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.799102 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.799079 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.800092 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.800068 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.800189 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.800169 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.800959 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.800709 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-grpc-tls\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.800959 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.800727 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-tls\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.802155 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.802131 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.807487 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.807467 2576 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7mts\" (UniqueName: \"kubernetes.io/projected/e25856e6-e50d-48d2-94a5-8966be78dca4-kube-api-access-k7mts\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.917983 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.917947 2576 generic.go:356] "Generic (PLEG): container finished" podID="896e7579-dea0-49ee-92c5-cddf258c6905" containerID="2b677e8a0d96833c3c46b8bec6f5ef89b7656fe6c548312cd60be33527fcd9a1" exitCode=0 Oct 31 13:03:49.918173 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.918043 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5fj4" event={"ID":"896e7579-dea0-49ee-92c5-cddf258c6905","Type":"ContainerDied","Data":"2b677e8a0d96833c3c46b8bec6f5ef89b7656fe6c548312cd60be33527fcd9a1"} Oct 31 13:03:49.921953 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.921880 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-7f7h9" event={"ID":"018c5e83-597e-4c20-9063-5287b4c332a2","Type":"ContainerStarted","Data":"4100818bf15102eb95fbfec4c568de630c2b60e65aa544a2636967272b19d34c"} Oct 31 13:03:49.922403 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.922383 2576 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:49.925484 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.925441 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64885574d7-l7vzc" event={"ID":"4d38ca15-69a2-4950-9bf5-c77868a86cfb","Type":"ContainerStarted","Data":"445f68b339fbb070e4bb7fa7195c1255bdc1bdc6ad63a526713ed16ff6c25231"} Oct 31 13:03:49.929261 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.929235 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" event={"ID":"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5","Type":"ContainerStarted","Data":"1d65e487d4fd6775be03b2fce59bb7853a9b578a055f8f71b52919908fa6cdfe"} Oct 31 13:03:49.931603 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.931563 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerStarted","Data":"153a7b0a80ec64f63310fca9ec410e95d6c16ec1526a12f84ae675e1622405ac"} Oct 31 13:03:49.936973 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.936937 2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:49.990634 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.989840 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-7f7h9" podStartSLOduration=66.780573892 podStartE2EDuration="1m11.989824471s" totalImagesPullingTime="5.209250579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:02:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:49.964081711 +0000 UTC m=+73.026879097" watchObservedRunningTime="2026-10-31 13:03:49.989824471 +0000 UTC m=+73.052621856" Oct 31 13:03:49.990634 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:49.989918 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-64885574d7-l7vzc" podStartSLOduration=2.16696897 podStartE2EDuration="6.989914734s" totalImagesPullingTime="4.822945764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:49.988208951 +0000 UTC m=+73.051006356" watchObservedRunningTime="2026-10-31 13:03:49.989914734 +0000 UTC m=+73.052712119" Oct 31 13:03:50.010501 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:50.010446 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" podStartSLOduration=1.572548048 podStartE2EDuration="3.010427192s" totalImagesPullingTime="1.437879144s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:50.008923043 +0000 UTC m=+73.071720429" watchObservedRunningTime="2026-10-31 13:03:50.010427192 +0000 UTC m=+73.073224590" Oct 31 13:03:50.236671 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:50.236362 2576 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57c987d54b-lc452"] Oct 31 13:03:50.239617 ip-10-0-128-41 kubenswrapper[2576]: W1031 13:03:50.239568 2576 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode25856e6_e50d_48d2_94a5_8966be78dca4.slice/crio-dffe519141b11297c380293f9e120bf018fd5526ffce945772356212df97a522 WatchSource:0}: Error finding container dffe519141b11297c380293f9e120bf018fd5526ffce945772356212df97a522: Status 404 returned error can't find the container with id dffe519141b11297c380293f9e120bf018fd5526ffce945772356212df97a522 Oct 31 13:03:50.946983 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:50.946938 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5fj4" event={"ID":"896e7579-dea0-49ee-92c5-cddf258c6905","Type":"ContainerStarted","Data":"e28f72fc825b65d1883e0706d8f0b5abc0be4d122ddf06097c078faece86582d"} Oct 31 13:03:50.947442 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:50.947042 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p5fj4" event={"ID":"896e7579-dea0-49ee-92c5-cddf258c6905","Type":"ContainerStarted","Data":"ffe27db3ac2f6b498245463ed1be925c9a20dee43bfdcbce8b61e24d091f2f1c"} Oct 31 13:03:50.949225 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:50.949186 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" event={"ID":"e25856e6-e50d-48d2-94a5-8966be78dca4","Type":"ContainerStarted","Data":"dffe519141b11297c380293f9e120bf018fd5526ffce945772356212df97a522"} Oct 31 13:03:50.951352 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:50.951325 2576 generic.go:356] "Generic (PLEG): container finished" podID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerID="5597b8699084216d1ebb3f8a5221731bf5b762337e5d25896e643d857b9a8089" exitCode=0 Oct 31 13:03:50.951701 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:50.951667 2576 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerDied","Data":"5597b8699084216d1ebb3f8a5221731bf5b762337e5d25896e643d857b9a8089"} Oct 31 13:03:50.974992 ip-10-0-128-41 kubenswrapper[2576]: I1031 13:03:50.974929 2576 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p5fj4" podStartSLOduration=2.33868841 podStartE2EDuration="3.974911956s" totalImagesPullingTime="1.554967802s" totalInitContainerRuntime="81.255744ms" isStatefulPod=true podCreationTimestamp="2026-10-31 13:03:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:50.973711534 +0000 UTC m=+74.036508919" watchObservedRunningTime="2026-10-31 13:03:50.974911956 +0000 UTC m=+74.037709342" Oct 31 13:03:51.759178 ip-10-0-128-41 systemd[1]: Stopping Kubernetes Kubelet... Oct 31 13:03:51.784219 ip-10-0-128-41 systemd[1]: kubelet.service: Deactivated successfully. Oct 31 13:03:51.784410 ip-10-0-128-41 systemd[1]: Stopped Kubernetes Kubelet. Oct 31 13:03:51.784628 ip-10-0-128-41 systemd[1]: kubelet.service: Consumed 3.832s CPU time. Oct 31 13:03:51.803458 ip-10-0-128-41 systemd[1]: Starting Kubernetes Kubelet... Oct 31 13:03:51.887945 ip-10-0-128-41 kubenswrapper[5532]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 13:03:51.887945 ip-10-0-128-41 kubenswrapper[5532]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 31 13:03:51.887945 ip-10-0-128-41 kubenswrapper[5532]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 13:03:51.888758 ip-10-0-128-41 kubenswrapper[5532]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 31 13:03:51.892479 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892361 5532 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:03:51.892479 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892475 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:03:51.892479 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892482 5532 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892487 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892492 5532 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892496 5532 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892500 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892505 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892509 5532 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892514 5532 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892518 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892523 5532 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892527 5532 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892536 5532 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892540 5532 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892544 5532 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892548 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892553 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892557 5532 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892561 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892565 5532 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:03:51.892728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892569 5532 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892573 5532 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892577 5532 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892582 5532 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892587 5532 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892608 5532 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892614 5532 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892620 5532 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892626 5532 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892631 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892639 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892645 5532 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892651 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892655 5532 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892660 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892665 5532 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892669 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892674 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892684 5532 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:03:51.893569 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892689 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892694 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892698 5532 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892703 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892709 5532 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892714 5532 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892718 5532 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892723 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892727 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892735 5532 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892742 5532 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892752 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892757 5532 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892762 5532 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892766 5532 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892771 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892775 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892780 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892784 5532 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:03:51.894503 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892789 5532 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892793 5532 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892798 5532 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892802 5532 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892808 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892818 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892823 5532 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892828 5532 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892832 5532 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892837 5532 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892842 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892846 5532 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892851 5532 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892856 5532 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892861 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892865 5532 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892870 5532 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892874 5532 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892884 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892889 5532 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:03:51.895325 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892893 5532 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892898 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892903 5532 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892907 5532 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892912 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892918 5532 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892923 5532 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892928 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892933 5532 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892944 5532 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892948 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892953 5532 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892957 5532 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892962 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892967 5532 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892972 5532 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892978 5532 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892983 5532 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892990 5532 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892994 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:03:51.896256 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.892999 5532 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.893012 5532 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.893017 5532 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.893022 5532 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.893027 5532 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.893032 5532 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.893036 5532 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895711 5532 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895723 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895729 5532 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895734 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895739 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895750 5532 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895755 5532 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895760 5532 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895764 5532 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895769 5532 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895773 5532 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895778 5532 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:03:51.897167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895783 5532 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895787 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895792 5532 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895797 5532 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895801 5532 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895805 5532 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895814 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895819 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895824 5532 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895829 5532 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895834 5532 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895839 5532 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895844 5532 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895848 5532 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895853 5532 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895858 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895862 5532 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895867 5532 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895876 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895880 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:03:51.897838 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895885 5532 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895889 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895897 5532 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895903 5532 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895908 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895913 5532 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895918 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895922 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895927 5532 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895931 5532 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895936 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895945 5532 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895949 5532 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895954 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895958 5532 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895965 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895969 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895974 5532 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895978 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:03:51.898432 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895983 5532 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895988 5532 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895992 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.895997 5532 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896006 5532 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896011 5532 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896016 5532 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896021 5532 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896026 5532 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896030 5532 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896034 5532 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896038 5532 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896043 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896047 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896051 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896056 5532 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896065 5532 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896069 5532 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896073 5532 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:03:51.899019 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896077 5532 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896082 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896087 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896091 5532 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896096 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896100 5532 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896105 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896110 5532 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896114 5532 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896119 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896128 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896132 5532 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896137 5532 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896142 5532 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896147 5532 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896151 5532 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896156 5532 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896160 5532 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896165 5532 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896171 5532 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:03:51.899582 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896178 5532 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896183 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896192 5532 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896196 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896202 5532 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896206 5532 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896210 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896214 5532 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896218 5532 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896224 5532 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896228 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896236 5532 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896240 5532 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896245 5532 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896253 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.896258 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897580 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897609 5532 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897617 5532 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897622 5532 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:03:51.900185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897630 5532 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897635 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897640 5532 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897644 5532 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897651 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897655 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897659 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897665 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897669 5532 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897673 5532 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897675 5532 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897678 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897681 5532 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897684 5532 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897686 5532 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897689 5532 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897692 5532 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897697 5532 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897702 5532 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:03:51.900843 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897705 5532 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897708 5532 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897712 5532 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897715 5532 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897717 5532 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897720 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897723 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897727 5532 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897729 5532 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897732 5532 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897735 5532 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897738 5532 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897740 5532 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897743 5532 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897745 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897748 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897751 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897753 5532 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897756 5532 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897758 5532 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:03:51.901398 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897761 5532 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897763 5532 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897766 5532 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897769 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897771 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897774 5532 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897776 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897779 5532 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897782 5532 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897785 5532 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897789 5532 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897793 5532 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897798 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897803 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897807 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897809 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897812 5532 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897815 5532 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897818 5532 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:03:51.901987 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897821 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897823 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897826 5532 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897829 5532 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897832 5532 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897835 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897838 5532 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897842 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897844 5532 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897847 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897850 5532 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897853 5532 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897856 5532 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897858 5532 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897861 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897863 5532 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897866 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897870 5532 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897874 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:03:51.902574 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897878 5532 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897881 5532 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897884 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897887 5532 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897889 5532 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897892 5532 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897895 5532 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897898 5532 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897901 5532 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897903 5532 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897906 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897909 5532 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897912 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897915 5532 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897917 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897920 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897923 5532 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897929 5532 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897933 5532 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897938 5532 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:03:51.903150 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897945 5532 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897949 5532 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897954 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897959 5532 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.897963 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898080 5532 flags.go:64] FLAG: --address="0.0.0.0" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898090 5532 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898096 5532 flags.go:64] FLAG: --anonymous-auth="true" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898101 5532 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898106 5532 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898110 5532 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898114 5532 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898118 5532 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898122 5532 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898125 5532 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898129 5532 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898133 5532 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898138 5532 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898144 5532 flags.go:64] FLAG: --cgroup-root="" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898149 5532 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898154 5532 flags.go:64] FLAG: --client-ca-file="" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898159 5532 flags.go:64] FLAG: --cloud-provider="external" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898165 5532 flags.go:64] FLAG: --cluster-dns="[]" Oct 31 13:03:51.903829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898170 5532 flags.go:64] FLAG: --cluster-domain="" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898173 5532 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898176 5532 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898179 5532 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898183 5532 flags.go:64] FLAG: --container-log-max-files="5" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898187 5532 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898190 5532 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898195 5532 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898198 5532 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898201 5532 flags.go:64] FLAG: --contention-profiling="false" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898204 5532 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898208 5532 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898211 5532 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898214 5532 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898218 5532 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898222 5532 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898225 5532 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898229 5532 flags.go:64] FLAG: --enable-load-reader="false" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898232 5532 flags.go:64] FLAG: --enable-server="true" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898235 5532 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898240 5532 flags.go:64] FLAG: --event-burst="100" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898243 5532 flags.go:64] FLAG: --event-qps="50" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898246 5532 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898250 5532 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 31 13:03:51.904488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898253 5532 flags.go:64] FLAG: --eviction-hard="" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898264 5532 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898268 5532 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898274 5532 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898280 5532 flags.go:64] FLAG: --eviction-soft="" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898285 5532 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898296 5532 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898302 5532 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898307 5532 flags.go:64] FLAG: --experimental-mounter-path="" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898312 5532 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898317 5532 flags.go:64] FLAG: --fail-swap-on="true" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898322 5532 flags.go:64] FLAG: --feature-gates="" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898329 5532 flags.go:64] FLAG: --file-check-frequency="20s" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898335 5532 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898340 5532 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898344 5532 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898348 5532 flags.go:64] FLAG: --healthz-port="10248" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898352 5532 flags.go:64] FLAG: --help="false" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898355 5532 flags.go:64] FLAG: --hostname-override="ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898358 5532 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898362 5532 flags.go:64] FLAG: --http-check-frequency="20s" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898364 5532 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898368 5532 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 31 13:03:51.905237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898372 5532 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898375 5532 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898378 5532 flags.go:64] FLAG: --image-service-endpoint="" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898380 5532 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898383 5532 flags.go:64] FLAG: --kube-api-burst="100" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898386 5532 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898390 5532 flags.go:64] FLAG: --kube-api-qps="50" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898393 5532 flags.go:64] FLAG: --kube-reserved="" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898396 5532 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898399 5532 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898402 5532 flags.go:64] FLAG: --kubelet-cgroups="" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898405 5532 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898408 5532 flags.go:64] FLAG: --lock-file="" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898411 5532 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898414 5532 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898417 5532 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898423 5532 flags.go:64] FLAG: --log-json-split-stream="false" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898426 5532 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898430 5532 flags.go:64] FLAG: --log-text-split-stream="false" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898433 5532 flags.go:64] FLAG: --logging-format="text" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898436 5532 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898440 5532 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898443 5532 flags.go:64] FLAG: --manifest-url="" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898446 5532 flags.go:64] FLAG: --manifest-url-header="" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898451 5532 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 31 13:03:51.906076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898455 5532 flags.go:64] FLAG: --max-open-files="1000000" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898459 5532 flags.go:64] FLAG: --max-pods="110" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898463 5532 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898466 5532 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898469 5532 flags.go:64] FLAG: --memory-manager-policy="None" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898472 5532 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898475 5532 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898478 5532 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898483 5532 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898500 5532 flags.go:64] FLAG: --node-status-max-images="50" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898504 5532 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898507 5532 flags.go:64] FLAG: --oom-score-adj="-999" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898510 5532 flags.go:64] FLAG: --pod-cidr="" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898516 5532 flags.go:64] FLAG: --pod-manifest-path="" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898519 5532 flags.go:64] FLAG: --pod-max-pids="-1" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898522 5532 flags.go:64] FLAG: --pods-per-core="0" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898525 5532 flags.go:64] FLAG: --port="10250" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898528 5532 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898531 5532 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-042410bbffce94d1b" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898534 5532 flags.go:64] FLAG: --qos-reserved="" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898537 5532 flags.go:64] FLAG: --read-only-port="10255" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898540 5532 flags.go:64] FLAG: --register-node="true" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898544 5532 flags.go:64] FLAG: --register-with-taints="" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898548 5532 flags.go:64] FLAG: --registry-burst="10" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898550 5532 flags.go:64] FLAG: --registry-qps="5" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898553 5532 flags.go:64] FLAG: --reserved-cpus="" Oct 31 13:03:51.906923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898556 5532 flags.go:64] FLAG: --reserved-memory="" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898563 5532 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898566 5532 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898569 5532 flags.go:64] FLAG: --rotate-certificates="false" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898572 5532 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898575 5532 flags.go:64] FLAG: --runonce="false" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898578 5532 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898581 5532 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898586 5532 flags.go:64] FLAG: --seccomp-default="false" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898589 5532 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898606 5532 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898611 5532 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898616 5532 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898621 5532 flags.go:64] FLAG: --storage-driver-password="root" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898625 5532 flags.go:64] FLAG: --storage-driver-secure="false" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898628 5532 flags.go:64] FLAG: --storage-driver-table="stats" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898632 5532 flags.go:64] FLAG: --storage-driver-user="root" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898635 5532 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898638 5532 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898642 5532 flags.go:64] FLAG: --system-cgroups="" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898645 5532 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898651 5532 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898654 5532 flags.go:64] FLAG: --tls-cert-file="" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898658 5532 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898662 5532 flags.go:64] FLAG: --tls-min-version="" Oct 31 13:03:51.907761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898665 5532 flags.go:64] FLAG: --tls-private-key-file="" Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898669 5532 flags.go:64] FLAG: --topology-manager-policy="none" Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898674 5532 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898679 5532 flags.go:64] FLAG: --topology-manager-scope="container" Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898684 5532 flags.go:64] FLAG: --v="2" Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898689 5532 flags.go:64] FLAG: --version="false" Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898694 5532 flags.go:64] FLAG: --vmodule="" Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898698 5532 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.898703 5532 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898856 5532 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898862 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898867 5532 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898873 5532 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898879 5532 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898883 5532 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898889 5532 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898898 5532 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898902 5532 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898905 5532 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898907 5532 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898910 5532 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898913 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:03:51.908545 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898915 5532 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898918 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898921 5532 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898924 5532 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898929 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898931 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898935 5532 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898940 5532 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898944 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898948 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898953 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898956 5532 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898959 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898962 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898964 5532 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898967 5532 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898970 5532 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898973 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898978 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:03:51.909263 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898980 5532 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898983 5532 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898986 5532 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898988 5532 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898991 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898994 5532 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.898998 5532 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899001 5532 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899005 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899007 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899010 5532 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899013 5532 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899015 5532 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899018 5532 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899020 5532 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899023 5532 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899026 5532 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899032 5532 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899035 5532 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:03:51.910099 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899037 5532 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899040 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899043 5532 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899046 5532 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899049 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899051 5532 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899054 5532 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899056 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899061 5532 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899065 5532 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899069 5532 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899074 5532 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899079 5532 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899081 5532 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899084 5532 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899087 5532 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899090 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899092 5532 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899095 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899098 5532 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:03:51.910769 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899101 5532 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899103 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899107 5532 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899110 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899113 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899115 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899118 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899120 5532 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899123 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899126 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899130 5532 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899133 5532 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899136 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899141 5532 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899147 5532 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899153 5532 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899156 5532 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899159 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899162 5532 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:03:51.911461 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899165 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899167 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899170 5532 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899173 5532 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899176 5532 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899180 5532 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899183 5532 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899186 5532 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899189 5532 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899191 5532 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899194 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899197 5532 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899200 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899203 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899205 5532 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899208 5532 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:03:51.912086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.899237 5532 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899281 5532 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899286 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899290 5532 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899293 5532 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899296 5532 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899299 5532 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899303 5532 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899306 5532 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899308 5532 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899311 5532 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899316 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899320 5532 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899324 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899327 5532 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899330 5532 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899333 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899336 5532 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899338 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899341 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:03:51.912575 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899345 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899349 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899355 5532 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899360 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899364 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899367 5532 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899369 5532 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899372 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899375 5532 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899379 5532 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899382 5532 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899385 5532 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899388 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899392 5532 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899396 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899398 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899401 5532 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899404 5532 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899406 5532 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:03:51.913171 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899410 5532 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899413 5532 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899416 5532 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899418 5532 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899421 5532 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899424 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899426 5532 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899429 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899431 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899434 5532 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899437 5532 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899439 5532 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899442 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899444 5532 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899447 5532 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899451 5532 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899454 5532 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899456 5532 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899459 5532 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899461 5532 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:03:51.913785 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899464 5532 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899467 5532 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899469 5532 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899474 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899478 5532 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899483 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899488 5532 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899493 5532 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899498 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899502 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899507 5532 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899511 5532 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899518 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899522 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899528 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899533 5532 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899537 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899542 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899545 5532 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899548 5532 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:03:51.914412 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899550 5532 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899553 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899556 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899558 5532 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899561 5532 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899564 5532 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899567 5532 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899571 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899574 5532 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899578 5532 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899581 5532 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899585 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899588 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899603 5532 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899606 5532 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899609 5532 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899613 5532 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899615 5532 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899618 5532 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:03:51.915149 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899621 5532 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899625 5532 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899628 5532 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899632 5532 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899635 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899639 5532 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899643 5532 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899645 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.899648 5532 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.903785 5532 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.903803 5532 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.903821 5532 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.903829 5532 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903890 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903896 5532 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903900 5532 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903903 5532 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903905 5532 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903908 5532 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:03:51.915732 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903911 5532 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903914 5532 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903917 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903919 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903923 5532 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903926 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903929 5532 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903932 5532 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903935 5532 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903938 5532 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903941 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903943 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903946 5532 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903949 5532 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903951 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903955 5532 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903957 5532 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903960 5532 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903963 5532 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903966 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:03:51.916266 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903968 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903971 5532 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903974 5532 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903979 5532 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903984 5532 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903987 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903990 5532 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903993 5532 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903995 5532 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.903998 5532 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904001 5532 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904003 5532 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904006 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904008 5532 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904011 5532 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904014 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904017 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904020 5532 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904023 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:03:51.916973 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904025 5532 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904028 5532 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904032 5532 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904034 5532 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904037 5532 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904040 5532 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904043 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904045 5532 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904049 5532 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904053 5532 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904057 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904062 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904067 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904070 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904073 5532 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904075 5532 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904078 5532 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904081 5532 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904085 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:03:51.917644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904088 5532 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904091 5532 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904094 5532 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904096 5532 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904099 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904102 5532 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904104 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904107 5532 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904110 5532 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904113 5532 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904116 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904119 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904122 5532 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904124 5532 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904127 5532 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904130 5532 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904132 5532 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904135 5532 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904138 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904140 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:03:51.918408 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904143 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904146 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904148 5532 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904152 5532 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904156 5532 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904159 5532 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904162 5532 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904166 5532 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904169 5532 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904172 5532 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904175 5532 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904179 5532 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904182 5532 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904185 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904188 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904191 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904195 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904199 5532 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904203 5532 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:03:51.919090 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904207 5532 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904212 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904215 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.904244 5532 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904406 5532 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904414 5532 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904417 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904420 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904423 5532 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904426 5532 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904429 5532 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904432 5532 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904436 5532 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904440 5532 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904443 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904447 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:03:51.919775 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904450 5532 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904453 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904456 5532 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904459 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904461 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904465 5532 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904467 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904470 5532 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904472 5532 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904476 5532 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904481 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904485 5532 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904490 5532 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904495 5532 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904499 5532 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904501 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904504 5532 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904507 5532 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904512 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904516 5532 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:03:51.920451 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904519 5532 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904522 5532 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904525 5532 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904528 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904530 5532 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904533 5532 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904536 5532 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904539 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904541 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904544 5532 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904547 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904549 5532 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904552 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904554 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904557 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904560 5532 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904565 5532 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904569 5532 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904574 5532 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904577 5532 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:03:51.921452 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904580 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904582 5532 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904585 5532 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904589 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904607 5532 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904611 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904629 5532 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904632 5532 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904635 5532 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904638 5532 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904640 5532 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904643 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904645 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904648 5532 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904652 5532 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904656 5532 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904660 5532 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904665 5532 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904668 5532 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:03:51.922305 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904671 5532 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904674 5532 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904676 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904679 5532 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904681 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904685 5532 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904689 5532 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904694 5532 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904698 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904701 5532 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904704 5532 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904707 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904711 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904714 5532 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904716 5532 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904719 5532 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904723 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904725 5532 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904728 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904730 5532 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:03:51.923127 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904733 5532 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904737 5532 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904741 5532 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904744 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904746 5532 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904749 5532 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904751 5532 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904754 5532 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904756 5532 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904759 5532 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904761 5532 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904764 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904767 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904769 5532 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904772 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:03:51.923981 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.904799 5532 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904838 5532 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904844 5532 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904847 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904850 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904852 5532 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904855 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904858 5532 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904860 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904863 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904865 5532 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904868 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904870 5532 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904873 5532 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904876 5532 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904879 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904882 5532 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904887 5532 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904891 5532 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 31 13:03:51.924765 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904895 5532 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904898 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904901 5532 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904903 5532 feature_gate.go:386] unrecognized feature gate: Example2 Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904906 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904911 5532 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904914 5532 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904917 5532 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904919 5532 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904922 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904924 5532 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904927 5532 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904930 5532 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904932 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904935 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904938 5532 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904940 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904943 5532 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904946 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 31 13:03:51.925629 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904948 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904951 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904953 5532 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904956 5532 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904958 5532 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904961 5532 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904963 5532 feature_gate.go:386] unrecognized feature gate: Example Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904966 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904969 5532 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904972 5532 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904977 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904981 5532 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904985 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904988 5532 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904990 5532 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904993 5532 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904996 5532 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.904998 5532 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905002 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 31 13:03:51.926798 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905005 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905007 5532 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905010 5532 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905013 5532 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905015 5532 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905018 5532 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905020 5532 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905023 5532 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905026 5532 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905029 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905031 5532 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905034 5532 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905037 5532 feature_gate.go:386] unrecognized feature gate: EVPN Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905039 5532 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905042 5532 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905044 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905048 5532 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905052 5532 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905057 5532 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905061 5532 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 31 13:03:51.927713 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905063 5532 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905066 5532 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905069 5532 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905072 5532 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905074 5532 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905077 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905079 5532 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905082 5532 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905086 5532 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905089 5532 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905092 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905096 5532 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905099 5532 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905102 5532 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905105 5532 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905107 5532 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905110 5532 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905112 5532 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905115 5532 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 31 13:03:51.928689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905117 5532 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905120 5532 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905123 5532 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905125 5532 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905128 5532 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905130 5532 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905133 5532 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905135 5532 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905140 5532 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905142 5532 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:51.905145 5532 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.905199 5532 server.go:957] "Client rotation is on, will bootstrap in background" Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.906386 5532 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.906453 5532 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.906706 5532 server.go:1014] "Starting client certificate rotation" Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.906832 5532 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 31 13:03:51.929558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.906893 5532 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-30 12:57:37 +0000 UTC" deadline="2028-08-17 01:48:13.76299812 +0000 UTC" Oct 31 13:03:51.930229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.906932 5532 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15732h44m21.856068046s" Oct 31 13:03:51.930229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.907305 5532 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 13:03:51.930229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.908686 5532 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 31 13:03:51.930229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.911088 5532 log.go:25] "Validated CRI v1 runtime API" Oct 31 13:03:51.930229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.913537 5532 log.go:25] "Validated CRI v1 image API" Oct 31 13:03:51.930229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.914467 5532 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 31 13:03:51.930229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.917063 5532 fs.go:135] Filesystem UUIDs: map[23873c31-c9aa-48a6-9630-edd3a87c7243:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a13bc9ed-8022-437b-a32c-b640aa1f4297:/dev/nvme0n1p3] Oct 31 13:03:51.930725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.917088 5532 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0db026a30d618fbaa7375a75aa984d06041c8e566648bb59d6e58e390eee588b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0db026a30d618fbaa7375a75aa984d06041c8e566648bb59d6e58e390eee588b/userdata/shm major:0 minor:318 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/153a7b0a80ec64f63310fca9ec410e95d6c16ec1526a12f84ae675e1622405ac/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/153a7b0a80ec64f63310fca9ec410e95d6c16ec1526a12f84ae675e1622405ac/userdata/shm major:0 minor:370 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1dcd59a48b4c83fd45e6638c9df05b23c118bbfdcc28b472a48f59b8bf4e3f90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1dcd59a48b4c83fd45e6638c9df05b23c118bbfdcc28b472a48f59b8bf4e3f90/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/270e54d1580e36f820e948bcc22a171e5adfbd5673f865b2c8897cf6859ff825/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/270e54d1580e36f820e948bcc22a171e5adfbd5673f865b2c8897cf6859ff825/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27bdeb9c4bd1d0a81f9456ba6c9c6332548e508993091dbfeffce7c648b3db76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27bdeb9c4bd1d0a81f9456ba6c9c6332548e508993091dbfeffce7c648b3db76/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/28b3c56c45fc5879b2fd1ec3d1cd8238d569fbebce0d4ff34da1b76e2d06b84e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/28b3c56c45fc5879b2fd1ec3d1cd8238d569fbebce0d4ff34da1b76e2d06b84e/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3045860abe8b538dad79b8b05d09704f6971fba3add3cf2465b74ce1928ba9cc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3045860abe8b538dad79b8b05d09704f6971fba3add3cf2465b74ce1928ba9cc/userdata/shm major:0 minor:286 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/365e4ac477b108f1e883c86733c6bf1d41c727c74d0fa2e18b97e0ec2abcbf90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/365e4ac477b108f1e883c86733c6bf1d41c727c74d0fa2e18b97e0ec2abcbf90/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4391b5b81d24ce2493d2f0b7b9b6ef23d8f563c80d320aa9092dd8c3fe4f09ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4391b5b81d24ce2493d2f0b7b9b6ef23d8f563c80d320aa9092dd8c3fe4f09ed/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4516dd758e1729253b19122fc3125003c52f4727674a48c15031e664419aee42/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4516dd758e1729253b19122fc3125003c52f4727674a48c15031e664419aee42/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/48e0fc447fee8b31eb4bf5419ebf18b2d26563d1df3528bbe6eeba897a599d65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/48e0fc447fee8b31eb4bf5419ebf18b2d26563d1df3528bbe6eeba897a599d65/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f93fe1ef3eec4e3c5b2f435d20d4de70c17d1dab9b05cb13fcec39e05d94004/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f93fe1ef3eec4e3c5b2f435d20d4de70c17d1dab9b05cb13fcec39e05d94004/userdata/shm major:0 minor:341 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61946c5f5c049e16bfb07ca0280dc21252438eb04d18b2e396b56e81454467f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61946c5f5c049e16bfb07ca0280dc21252438eb04d18b2e396b56e81454467f8/userdata/shm major:0 minor:339 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7113b20c24c9ecc50668af315409491a04bf884757624db5c480ebdf7e55509c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7113b20c24c9ecc50668af315409491a04bf884757624db5c480ebdf7e55509c/userdata/shm major:0 minor:307 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/881b5c111023e271fe18bd5f38d10e6785b51aa743839d14c6155c1cef8a4fdb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/881b5c111023e271fe18bd5f38d10e6785b51aa743839d14c6155c1cef8a4fdb/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89f3a55866b70c818d5c596220497aa026d37c6c0afe9c19d4852c7771fb6673/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89f3a55866b70c818d5c596220497aa026d37c6c0afe9c19d4852c7771fb6673/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c6b18eec02cd699f5ce009b11c8d6926a0fb8f98e3883d83d5bea81d45f9abd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c6b18eec02cd699f5ce009b11c8d6926a0fb8f98e3883d83d5bea81d45f9abd/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9d22eaca31a972fcd4c52990e3e01be055e6cbeedb68d4d20515d92c0ef46b79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9d22eaca31a972fcd4c52990e3e01be055e6cbeedb68d4d20515d92c0ef46b79/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e2e00a5cedbfc0ca16983e327195326fb3f7628e80e8fb45e7f79c479c139cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e2e00a5cedbfc0ca16983e327195326fb3f7628e80e8fb45e7f79c479c139cf/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9eedaf1a1fe55c6516bdcc356943d841e84ed7bc35fb6208aa3f7aead2eb62c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9eedaf1a1fe55c6516bdcc356943d841e84ed7bc35fb6208aa3f7aead2eb62c2/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c75cc3474d609218febf8d3a1929a443940db8c99210ddf4b15a4bc4bfab4ca5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c75cc3474d609218febf8d3a1929a443940db8c99210ddf4b15a4bc4bfab4ca5/userdata/shm major:0 minor:303 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2078686775b9649bc6bc06bb416b4930cdc01332e3a585abd56a575d299d979/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2078686775b9649bc6bc06bb416b4930cdc01332e3a585abd56a575d299d979/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d931074a761f4f450c2030c9edca2919d06d28e256e68d606a39bd28608a996e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d931074a761f4f450c2030c9edca2919d06d28e256e68d606a39bd28608a996e/userdata/shm major:0 minor:217 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dffe519141b11297c380293f9e120bf018fd5526ffce945772356212df97a522/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dffe519141b11297c380293f9e120bf018fd5526ffce945772356212df97a522/userdata/shm major:0 minor:398 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4ee2b25d23199c272ed6f880b4618f67728b41aaa3a25d3ae9c230aa91e9af1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4ee2b25d23199c272ed6f880b4618f67728b41aaa3a25d3ae9c230aa91e9af1/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed98c2b2756f3de31628d4bf91d8ca9a7314abd9d9b3018841dee14b910e30a2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed98c2b2756f3de31628d4bf91d8ca9a7314abd9d9b3018841dee14b910e30a2/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0104a2a9-918a-40bf-9ece-bfa7360a8d36/volumes/kubernetes.io~projected/kube-api-access-jm6n8:{mountpoint:/var/lib/kubelet/pods/0104a2a9-918a-40bf-9ece-bfa7360a8d36/volumes/kubernetes.io~projected/kube-api-access-jm6n8 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/018c5e83-597e-4c20-9063-5287b4c332a2/volumes/kubernetes.io~projected/kube-api-access-n54p8:{mountpoint:/var/lib/kubelet/pods/018c5e83-597e-4c20-9063-5287b4c332a2/volumes/kubernetes.io~projected/kube-api-access-n54p8 major:0 minor:302 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05baffb8-cba0-444a-989e-dc5df0c7bcd6/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/05baffb8-cba0-444a-989e-dc5df0c7bcd6/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05baffb8-cba0-444a-989e-dc5df0c7bcd6/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/05baffb8-cba0-444a-989e-dc5df0c7bcd6/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05baffb8-cba0-444a-989e-dc5df0c7bcd6/volumes/kubernetes.io~projected/kube-api-access-slrrr:{mountpoint:/var/lib/kubelet/pods/05baffb8-cba0-444a-989e-dc5df0c7bcd6/volumes/kubernetes.io~projected/kube-api-access-slrrr major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19180e90-ffed-457b-9d2e-92645efe3491/volumes/kubernetes.io~projected/kube-api-access-v79lg:{mountpoint:/var/lib/kubelet/pods/19180e90-ffed-457b-9d2e-92645efe3491/volumes/kubernetes.io~projected/kube-api-access-v79lg major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/25233e95-abc9-4f04-b7b2-65933229d471/volumes/kubernetes.io~secret/tls-certificates:{mountpoint:/var/lib/kubelet/pods/25233e95-abc9-4f04-b7b2-65933229d471/volumes/kubernetes.io~secret/tls-certificates major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2aee367b-5b87-4bfe-9620-5adf75e25b63/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2aee367b-5b87-4bfe-9620-5adf75e25b63/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:301 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~empty-dir/config-out major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~projected/kube-api-access-xdn2q:{mountpoint:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~projected/kube-api-access-xdn2q major:0 minor:367 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~projected/tls-assets major:0 minor:366 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/config-volume major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:365 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/web-config major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5/volumes/kubernetes.io~projected/kube-api-access-vxr4p:{mountpoint:/var/lib/kubelet/pods/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5/volumes/kubernetes.io~projected/kube-api-access-vxr4p major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:330 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d38ca15-69a2-4950-9bf5-c77868a86cfb/volumes/kubernetes.io~projected/kube-api-access-ml8wx:{mountpoint:/var/lib/kubelet/pods/4d38ca15-69a2-4950-9bf5-c77868a86cfb/volumes/kubernetes.io~projected/kube-api-access-ml8wx major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d38ca15-69a2-4950-9bf5-c77868a86cfb/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/4d38ca15-69a2-4950-9bf5-c77868a86cfb/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d38ca15-69a2-4950-9bf5-c77868a86cfb/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/4d38ca15-69a2-4950-9bf5-c77868a86cfb/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54a25982-6507-405a-8e10-81d965ab7f4f/volumes/kubernetes.io~projected/kube-api-access-nq4sr:{mountpoint:/var/lib/kubelet/pods/54a25982-6507-405a-8e10-81d965ab7f4f/volumes/kubernetes.io~projected/kube-api-access-nq4sr major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70a6eabd-c790-46fa-a9f1-cff156624c2f/volumes/kubernetes.io~projected/kube-api-access-cw7j7:{mountpoint:/var/lib/kubelet/pods/70a6eabd-c790-46fa-a9f1-cff156624c2f/volumes/kubernetes.io~projected/kube-api-access-cw7j7 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70a6eabd-c790-46fa-a9f1-cff156624c2f/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/70a6eabd-c790-46fa-a9f1-cff156624c2f/volumes/kubernetes.io~secret/metrics-certs major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/70ea50fe-3024-4a82-84a5-5c236706402b/volumes/kubernetes.io~projected/kube-api-access-7qj4l:{mountpoint:/var/lib/kubelet/pods/70ea50fe-3024-4a82-84a5-5c236706402b/volumes/kubernetes.io~projected/kube-api-access-7qj4l major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5/volumes/kubernetes.io~projected/kube-api-access-sx88t:{mountpoint:/var/lib/kubelet/pods/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5/volumes/kubernetes.io~projected/kube-api-access-sx88t major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/896e7579-dea0-49ee-92c5-cddf258c6905/volumes/kubernetes.io~projected/kube-api-access-tm6fc:{mountpoint:/var/lib/kubelet/pods/896e7579-dea0-49ee-92c5-cddf258c6905/volumes/kubernetes.io~projected/kube-api-access-tm6fc major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/896e7579-dea0-49ee-92c5-cddf258c6905/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/896e7579-dea0-49ee-92c5-cddf258c6905/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:336 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/896e7579-dea0-49ee-92c5-cddf258c6905/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/896e7579-dea0-49ee-92c5-cddf258c6905/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0b84e1d-390d-434d-b309-9440ec8195b6/volumes/kubernetes.io~projected/kube-api-access-q88wd:{mountpoint:/var/lib/kubelet/pods/a0b84e1d-390d-434d-b309-9440ec8195b6/volumes/kubernetes.io~projected/kube-api-access-q88wd major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a0b84e1d-390d-434d-b309-9440ec8195b6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/a0b84e1d-390d-434d-b309-9440ec8195b6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c913d3bd-b2b9-42d0-a02b-e0474c15e14c/volumes/kubernetes.io~projected/kube-api-access-lkj5f:{mountpoint:/var/lib/kubelet/pods/c913d3bd-b2b9-42d0-a02b-e0474c15e14c/volumes/kubernetes.io~projected/kube-api-access-lkj5f major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ccf935cb-694a-4d54-b7a3-f0be4bc1212c/volumes/kubernetes.io~projected/kube-api-access-jfnsx:{mountpoint:/var/lib/kubelet/pods/ccf935cb-694a-4d54-b7a3-f0be4bc1212c/volumes/kubernetes.io~projected/kube-api-access-jfnsx major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2eab6ed-c191-4b42-899c-e816ab8d4a0b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d2eab6ed-c191-4b42-899c-e816ab8d4a0b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2eab6ed-c191-4b42-899c-e816ab8d4a0b/volumes/kubernetes.io~projected/kube-api-access-65x7x:{mountpoint:/var/lib/kubelet/pods/d2eab6ed-c191-4b42-899c-e816ab8d4a0b/volumes/kubernetes.io~projected/kube-api-access-65x7x major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d2eab6ed-c191-4b42-899c-e816ab8d4a0b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d2eab6ed-c191-4b42-899c-e816ab8d4a0b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dc1ff283-237c-45f3-a90f-88cb75bcb863/volumes/kubernetes.io~projected/kube-api-access-nr7hd:{mountpoint:/var/lib/kubelet/pods/dc1ff283-237c-45f3-a90f-88cb75bcb863/volumes/kubernetes.io~projected/kube-api-access-nr7hd major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~projected/kube-api-access-k7mts:{mountpoint:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~projected/kube-api-access-k7mts major:0 minor:395 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:392 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:394 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e45e4264-8ba9-4ec4-be49-ccc8d26472f1/volumes/kubernetes.io~projected/kube-api-access-rg99f:{mountpoint:/var/lib/kubelet/pods/e45e4264-8ba9-4ec4-be49-ccc8d26472f1/volumes/kubernetes.io~projected/kube-api-access-rg99f major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e507f475-28b0-43c1-86b1-5ee3e4c4dad4/volumes/kubernetes.io~projected/kube-api-access-4sfsw:{mountpoint:/var/lib/kubelet/pods/e507f475-28b0-43c1-86b1-5ee3e4c4dad4/volumes/kubernetes.io~projected/kube-api-access-4sfsw major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e507f475-28b0-43c1-86b1-5ee3e4c4dad4/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/e507f475-28b0-43c1-86b1-5ee3e4c4dad4/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea0c07d8-f966-4acb-aea6-1ec8682b4877/volumes/kubernetes.io~projected/kube-api-access-8bt8n:{mountpoint:/var/lib/kubelet/pods/ea0c07d8-f966-4acb-aea6-1ec8682b4877/volumes/kubernetes.io~projected/kube-api-access-8bt8n major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ea0c07d8-f966-4acb-aea6-1ec8682b4877/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/ea0c07d8-f966-4acb-aea6-1ec8682b4877/volumes/kubernetes.io~secret/metrics-tls major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1b612e1-018b-47d0-880a-cb879202e312/volumes/kubernetes.io~projected/kube-api-access-5dk65:{mountpoint:/var/lib/kubelet/pods/f1b612e1-018b-47d0-880a-cb879202e312/volumes/kubernetes.io~projected/kube-api-access-5dk65 major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1b612e1-018b-47d0-880a-cb879202e312/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/f1b612e1-018b-47d0-880a-cb879202e312/volumes/kubernetes.io~secret/signing-key major:0 minor:213 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/6d1f6c23ea5cbe9b60dafffe3d80259e9c97eb0739258655e77abceb28580ddc/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f0a32f38c3352a91719fdbdd5b0f5af0cce8e5c6c4f411431de4a4e780d462d8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/a42ee6179a38f602fa2eed3823176ccbb8f32789a3c72220911900f8a1634b87/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/0dec488359827c228482b89a1cf49fafcf42046276edf2c1e1de71e047182fca/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/5bdeda4c1dc6d7be6f3ef4bfdfb903026b017ca179428badb78b4b0add690e00/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/da1772bcd2a8614863a2f378e715111db14976db023189d192d39ecfe7e81b72/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f04c418b074cc2731a3ef6721b886de3f1c185dc9e97a9d0459e92c3b81f7724/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b76c62ea9313e2d19817d415797c7c3ee75797ad788d806abf80e92ae7dc1292/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/94b5c4032e92872f2c9b0e354df54c98e02196dd4f04a8e4bab3ab81a64d08e5/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/b46f9df5afb74d8a79e20e594a4ebc5a452179b9936d09bdfd4c97806a275f56/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/ba60dc168d854836b19693eda1ec6365993972fef25020ff729779a849d893b1/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/0aa1dd1101cbd0b267497d4f0208f4d5fc12fc1533a110a42977cd74c6ad1ee2/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ae1b0612996f03f3dc5840dbcf3da9107730013d04f672cce75a8161a9531cbf/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/cd26847d8829385df08f121381eff9d55e72b48b38625dbac67d7bdf98e1292e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/3b2cfc4b37cf883c3016fa2a53d23880c1e1d7b5618dc7c020fedfc182e7d3ea/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f9b0f5049fb26e6f327c3449d28f697bcddb4ce04ec40ac2c8b7d17584d0173f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/f45b0dc50c498b25bc2cecf3be33af4bfc84443d2e5cd211bd240b8a4b2967c6/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/c4fb89ec2beaaed335848672fc0216d2b3ee6353e3112cb419c1c2fe77622df2/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/f80118f751c34d44c1a961a93914f5b430030ad536bba4f60fb729cf6ec9679b/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/0657204c322767811ad1e7a58498121ce619325f50bc96a31763e12a2590eb58/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/020078638efaffc75cde5946cfe82871154a0a9e41e216b0329717b283b4dfbc/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/e5fe8eb2b98626e266a48877150d1cddb34e5a7dcf0c5d65f41c3b919b46e355/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/d12d5485722c89858cebb2acc1086cbc8e52075b73e30f9fa0f040d81e1cefd3/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-223:{mountpoint:/var/lib/containers/storage/overlay/977a1a0be610f519552fada8c28b5d835e560b09432581115faa5084f13dcc65/merged major:0 minor:223 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/75beb9920513dcaedfe20a4cc282bcc39df6f3169245ab07932dafdf4181df65/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/7e4285ec53aa55b899830c3025f1565ce41f76215005c886735febcafc664182/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/525b03419160bbd290936f3b90b865dc05723038feb531e65eb00a07db09b82c/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/20aeaac7737957103ffbd615202ef45f29808ff03bfb9ea6dc20750efd52167d/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/85927e19c8b0a21cdcb2093bd592bce43f75c80bd465586a68fdbc8073e4d418/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/5e6b041781cb9706b698697e8c2add23c5fd7fadab5e37350462bb9f671157b2/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/73100bb39669be789ff4e2ce77358c8f6acb2314f6a6e6dafaea38c26fa69db2/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/67b2e9c336a5e30d932cd03ce96cee97a2b8a03b8e23b9da2f30b251f21003c7/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/c8401bcbe25ccb7d5543cbe73212b2928967053d1012a3ea3c38b8436ea1ac74/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/67751c084a9d8e97cc50fed437fca73cd4d0f8b637221a0e93e5bbcc6b7aa7d6/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/89a59e7f204dbe4c78c57e58317e95b0e81814c977a9632d067288efcefa1ead/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/c48a6e35dc4078bac321397b060ebdb292800f79a6647c21bcfc0524af35ca26/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/b6adde3d21e84418093520cf8a3d5602a78cea70ffe0e306e7d314768aca512c/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/a567502630d6edb990d5590923fe0106efaaeda7a36b227854572b6f667f291c/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/802609d5db57a435bbca11f1c80b5f831d39c447d16878d87a19592d9cbcec36/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/163aecaa05116312eba7baa8280c2ff99fe0f62768508ba2b682eca80085eb1a/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/bf6867b747ec82635412a2f0288e8014a1450ea7deb644fc2dbf7e2ae69d9007/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/5e42bd63509fc1d40fbecf1ac50a6ae99b407fc15e3d2117cf45ac5201f9485e/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/f38155c44d2f0814dac48634f19f3ced3fc4d262f14d682806d77a916314ffd8/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/cd2d727665b5ebeada7b8d0e35c2ef06842d9ce79e250849b32f74fb62382a4b/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/e3f1e6d4354d14cdb49915260719e9be5f302edd722ee792e827c6a99bcf391c/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/d55c14b6c3a6d99f0f59814285c75c9780d0b340cc0e929ef50d73e5d02d6e78/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/cfcfab08bb8579f2978e261296854a796ca30fc1dd401f538505843a2a404a6c/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-347:{mountpoint:/var/lib/containers/storage/overlay/b6ef14aeffcd5c4386e99d23deb213033564775015aa2ead71519244ea9cb539/merged major:0 minor:347 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/470c17367b6e19d27f02857e52a671191253c090cc02ee7e1d1b6070838cbb06/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/10dd06a3934eb1d102a287846a4564359a25a5f7ac3a2ff5a7efda1a57c57d82/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/aff418f3a2c71f785a81479411390f6222be248081b08c96a32805f7e957704c/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-378:{mountpoint:/var/lib/containers/storage/overlay/4844d2b2a4720dc120d421624caeb9966a91e67127ed5c09f83890c1d5eadf44/merged major:0 minor:378 fsType:overlay blockSize:0} overlay_0-380:{mountpoint:/var/lib/containers/storage/overlay/d8e301cbd3e6643302112d59634c4369cf9443abc0d913ab5d6457175a36cd85/merged major:0 minor:380 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/0fe2d8501f22fe1a6d89b0c05ae88b497af482f260c7992543e00afec8db2558/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/be4296e7fe6bf79ffe29ba95209876df29709ed8c02b59a1aa14d10dfd074974/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-405:{mountpoint:/var/lib/containers/storage/overlay/ad40f575f600953c00fc791915c34b11e29c450b476d1e05a1ca83db6493c1cf/merged major:0 minor:405 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/17a81971816c6468612751918fe7eda2883c821ed6b795dc3a234c2633b56161/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/8709d38914144ac78f6ff9a10c8db3866db5e70f2bf830afd9d18382ca646915/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/cde5520d3afc2af85408c8f0adabd50766a3d19385826168804786879a4c65c8/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/d8fba0f0b2fc3f057bcc722699ad58031ce04cd8bf85071dc578e0278eda86d5/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/2099c0eb02632602f75b112875f89e97d0d3aa9ae919ac9301e5614bd0cc6bf3/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/f17598728cc97e2a52b3fec952dd336a317dd877c8542eec36752658c0fb72f4/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/b70ef06c0fdaa480af60e30fb985dd3fb0f5ef1fa04557cbca4a61b53179fc80/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/e760992358e7f8745ef9d0b59c1752726759ae0fe49e7e59c20fed0ef33ec770/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/1dd1c2cf31ccfa9d1c642e74edccee726aa26216b569307c15d9760652cc1438/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/586413c746a53130fa2dee57e033baa8e7d67fe76e9b640f2b65a097dda380b9/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d755c31f3d8f8b15c925613d65fd25dced44f9115e44cdd1f90709fae079514e/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 31 13:03:51.931130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.930189 5532 manager.go:217] Machine: {Timestamp:2026-10-31 13:03:51.929341073 +0000 UTC m=+0.075084899 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100283 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec24479514f8e05737fad8540417e159 SystemUUID:ec244795-14f8-e057-37fa-d8540417e159 BootID:7b835058-1da9-4c2c-9948-d949b1b39189 Filesystems:[{Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f93fe1ef3eec4e3c5b2f435d20d4de70c17d1dab9b05cb13fcec39e05d94004/userdata/shm DeviceMajor:0 DeviceMinor:341 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/70ea50fe-3024-4a82-84a5-5c236706402b/volumes/kubernetes.io~projected/kube-api-access-7qj4l DeviceMajor:0 DeviceMinor:285 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:392 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9d22eaca31a972fcd4c52990e3e01be055e6cbeedb68d4d20515d92c0ef46b79/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-223 DeviceMajor:0 DeviceMinor:223 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/365e4ac477b108f1e883c86733c6bf1d41c727c74d0fa2e18b97e0ec2abcbf90/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:330 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/153a7b0a80ec64f63310fca9ec410e95d6c16ec1526a12f84ae675e1622405ac/userdata/shm DeviceMajor:0 DeviceMinor:370 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-378 DeviceMajor:0 DeviceMinor:378 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:359 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:360 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9eedaf1a1fe55c6516bdcc356943d841e84ed7bc35fb6208aa3f7aead2eb62c2/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/48e0fc447fee8b31eb4bf5419ebf18b2d26563d1df3528bbe6eeba897a599d65/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/896e7579-dea0-49ee-92c5-cddf258c6905/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:335 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:355 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:394 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4516dd758e1729253b19122fc3125003c52f4727674a48c15031e664419aee42/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/70a6eabd-c790-46fa-a9f1-cff156624c2f/volumes/kubernetes.io~projected/kube-api-access-cw7j7 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed98c2b2756f3de31628d4bf91d8ca9a7314abd9d9b3018841dee14b910e30a2/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4391b5b81d24ce2493d2f0b7b9b6ef23d8f563c80d320aa9092dd8c3fe4f09ed/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:365 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/018c5e83-597e-4c20-9063-5287b4c332a2/volumes/kubernetes.io~projected/kube-api-access-n54p8 DeviceMajor:0 DeviceMinor:302 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:364 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/54a25982-6507-405a-8e10-81d965ab7f4f/volumes/kubernetes.io~projected/kube-api-access-nq4sr DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e2e00a5cedbfc0ca16983e327195326fb3f7628e80e8fb45e7f79c479c139cf/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/896e7579-dea0-49ee-92c5-cddf258c6905/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:336 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:362 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/05baffb8-cba0-444a-989e-dc5df0c7bcd6/volumes/kubernetes.io~projected/kube-api-access-slrrr DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-347 DeviceMajor:0 DeviceMinor:347 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2eab6ed-c191-4b42-899c-e816ab8d4a0b/volumes/kubernetes.io~projected/kube-api-access-65x7x DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0b84e1d-390d-434d-b309-9440ec8195b6/volumes/kubernetes.io~projected/kube-api-access-q88wd DeviceMajor:0 DeviceMinor:216 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27bdeb9c4bd1d0a81f9456ba6c9c6332548e508993091dbfeffce7c648b3db76/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dffe519141b11297c380293f9e120bf018fd5526ffce945772356212df97a522/userdata/shm DeviceMajor:0 DeviceMinor:398 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5/volumes/kubernetes.io~projected/kube-api-access-sx88t DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1b612e1-018b-47d0-880a-cb879202e312/volumes/kubernetes.io~projected/kube-api-access-5dk65 DeviceMajor:0 DeviceMinor:214 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4ee2b25d23199c272ed6f880b4618f67728b41aaa3a25d3ae9c230aa91e9af1/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/881b5c111023e271fe18bd5f38d10e6785b51aa743839d14c6155c1cef8a4fdb/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/dc1ff283-237c-45f3-a90f-88cb75bcb863/volumes/kubernetes.io~projected/kube-api-access-nr7hd DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/f1b612e1-018b-47d0-880a-cb879202e312/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:213 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d931074a761f4f450c2030c9edca2919d06d28e256e68d606a39bd28608a996e/userdata/shm DeviceMajor:0 DeviceMinor:217 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c75cc3474d609218febf8d3a1929a443940db8c99210ddf4b15a4bc4bfab4ca5/userdata/shm DeviceMajor:0 DeviceMinor:303 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d38ca15-69a2-4950-9bf5-c77868a86cfb/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:316 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea0c07d8-f966-4acb-aea6-1ec8682b4877/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:264 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3045860abe8b538dad79b8b05d09704f6971fba3add3cf2465b74ce1928ba9cc/userdata/shm DeviceMajor:0 DeviceMinor:286 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2aee367b-5b87-4bfe-9620-5adf75e25b63/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:301 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/05baffb8-cba0-444a-989e-dc5df0c7bcd6/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~projected/kube-api-access-xdn2q DeviceMajor:0 DeviceMinor:367 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c6b18eec02cd699f5ce009b11c8d6926a0fb8f98e3883d83d5bea81d45f9abd/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2078686775b9649bc6bc06bb416b4930cdc01332e3a585abd56a575d299d979/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2eab6ed-c191-4b42-899c-e816ab8d4a0b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e507f475-28b0-43c1-86b1-5ee3e4c4dad4/volumes/kubernetes.io~projected/kube-api-access-4sfsw DeviceMajor:0 DeviceMinor:201 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ccf935cb-694a-4d54-b7a3-f0be4bc1212c/volumes/kubernetes.io~projected/kube-api-access-jfnsx DeviceMajor:0 DeviceMinor:215 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:363 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89f3a55866b70c818d5c596220497aa026d37c6c0afe9c19d4852c7771fb6673/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:391 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ea0c07d8-f966-4acb-aea6-1ec8682b4877/volumes/kubernetes.io~projected/kube-api-access-8bt8n DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d38ca15-69a2-4950-9bf5-c77868a86cfb/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:315 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/896e7579-dea0-49ee-92c5-cddf258c6905/volumes/kubernetes.io~projected/kube-api-access-tm6fc DeviceMajor:0 DeviceMinor:337 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/05baffb8-cba0-444a-989e-dc5df0c7bcd6/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:334 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:386 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d2eab6ed-c191-4b42-899c-e816ab8d4a0b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a0b84e1d-390d-434d-b309-9440ec8195b6/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:239 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/28b3c56c45fc5879b2fd1ec3d1cd8238d569fbebce0d4ff34da1b76e2d06b84e/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/70a6eabd-c790-46fa-a9f1-cff156624c2f/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:296 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5/volumes/kubernetes.io~projected/kube-api-access-vxr4p DeviceMajor:0 DeviceMinor:338 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61946c5f5c049e16bfb07ca0280dc21252438eb04d18b2e396b56e81454467f8/userdata/shm DeviceMajor:0 DeviceMinor:339 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/25233e95-abc9-4f04-b7b2-65933229d471/volumes/kubernetes.io~secret/tls-certificates DeviceMajor:0 DeviceMinor:281 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d38ca15-69a2-4950-9bf5-c77868a86cfb/volumes/kubernetes.io~projected/kube-api-access-ml8wx DeviceMajor:0 DeviceMinor:317 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:393 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~projected/kube-api-access-k7mts DeviceMajor:0 DeviceMinor:395 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e45e4264-8ba9-4ec4-be49-ccc8d26472f1/volumes/kubernetes.io~projected/kube-api-access-rg99f DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/270e54d1580e36f820e948bcc22a171e5adfbd5673f865b2c8897cf6859ff825/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:363 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/e25856e6-e50d-48d2-94a5-8966be78dca4/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:390 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-405 DeviceMajor:0 DeviceMinor:405 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/19180e90-ffed-457b-9d2e-92645efe3491/volumes/kubernetes.io~projected/kube-api-access-v79lg DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7113b20c24c9ecc50668af315409491a04bf884757624db5c480ebdf7e55509c/userdata/shm DeviceMajor:0 DeviceMinor:307 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0db026a30d618fbaa7375a75aa984d06041c8e566648bb59d6e58e390eee588b/userdata/shm DeviceMajor:0 DeviceMinor:318 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:366 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-380 DeviceMajor:0 DeviceMinor:380 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1dcd59a48b4c83fd45e6638c9df05b23c118bbfdcc28b472a48f59b8bf4e3f90/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:361 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/0104a2a9-918a-40bf-9ece-bfa7360a8d36/volumes/kubernetes.io~projected/kube-api-access-jm6n8 DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c913d3bd-b2b9-42d0-a02b-e0474c15e14c/volumes/kubernetes.io~projected/kube-api-access-lkj5f DeviceMajor:0 DeviceMinor:208 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e507f475-28b0-43c1-86b1-5ee3e4c4dad4/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0db026a30d618fb MacAddress:2e:6e:8a:8a:98:18 Speed:10000 Mtu:8901} {Name:153a7b0a80ec64f MacAddress:ee:b7:1e:9b:07:a4 Speed:10000 Mtu:8901} {Name:1dcd59a48b4c83f MacAddress:3e:c6:8e:75:68:bc Speed:10000 Mtu:8901} {Name:27bdeb9c4bd1d0a MacAddress:7e:8e:b5:0c:0a:33 Speed:10000 Mtu:8901} {Name:28b3c56c45fc587 MacAddress:16:e7:dc:a5:0d:8e Speed:10000 Mtu:8901} {Name:3045860abe8b538 MacAddress:d6:90:6d:59:74:9c Speed:10000 Mtu:8901} {Name:365e4ac477b108f MacAddress:de:2c:da:94:19:cd Speed:10000 Mtu:8901} {Name:61946c5f5c049e1 MacAddress:4e:35:82:59:43:20 Speed:10000 Mtu:8901} {Name:7113b20c24c9ecc MacAddress:f2:1f:e4:9e:b3:8d Speed:10000 Mtu:8901} {Name:9d22eaca31a972f MacAddress:06:f8:9c:52:e7:43 Speed:10000 Mtu:8901} {Name:9e2e00a5cedbfc0 MacAddress:d6:a7:11:65:47:91 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:fc:18:43:91 Speed:0 Mtu:9001} {Name:br-int MacAddress:1a:1f:8d:16:4f:f0 Speed:0 Mtu:8901} {Name:c75cc3474d60921 MacAddress:c6:a3:c9:f2:eb:5e Speed:10000 Mtu:8901} {Name:d931074a761f4f4 MacAddress:c2:20:1f:c7:04:4f Speed:10000 Mtu:8901} {Name:dffe519141b1129 MacAddress:aa:81:71:47:46:87 Speed:10000 Mtu:8901} {Name:e4ee2b25d23199c MacAddress:b2:08:ef:2d:d0:7b Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:fc:18:43:91 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:36:d2:c1:74:aa:df Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:a6:60:5d:dd:63:60 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 31 13:03:51.931130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.930980 5532 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 31 13:03:51.931130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.931052 5532 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 31 13:03:51.931485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.931449 5532 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 31 13:03:51.931670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.931489 5532 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-41.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 31 13:03:51.931733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.931674 5532 topology_manager.go:143] "Creating topology manager with none policy" Oct 31 13:03:51.931733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.931682 5532 container_manager_linux.go:311] "Creating device plugin manager" Oct 31 13:03:51.931733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.931692 5532 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 13:03:51.931733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.931712 5532 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 31 13:03:51.931846 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.931840 5532 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 31 13:03:51.932053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.932040 5532 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 31 13:03:51.932196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.932184 5532 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 31 13:03:51.932231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.932225 5532 kubelet.go:493] "Attempting to sync node with API server" Oct 31 13:03:51.932259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.932239 5532 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 31 13:03:51.932259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.932256 5532 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 31 13:03:51.932309 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.932267 5532 kubelet.go:405] "Adding apiserver pod source" Oct 31 13:03:51.932309 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.932276 5532 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 31 13:03:51.933268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.933252 5532 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 13:03:51.933268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.933266 5532 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 13:03:51.933834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.933819 5532 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 31 13:03:51.949392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.949372 5532 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 31 13:03:51.949392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.949390 5532 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 31 13:03:51.949571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.949518 5532 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 31 13:03:51.949790 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.949774 5532 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 31 13:03:51.949870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.949814 5532 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 31 13:03:51.949979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.949965 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 31 13:03:51.950038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.949994 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 31 13:03:51.950038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.950010 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 31 13:03:51.950038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.950021 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 31 13:03:51.950038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.950031 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 31 13:03:51.950205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.950041 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 31 13:03:51.950205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.950050 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 31 13:03:51.950205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.950059 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 31 13:03:51.950205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.950070 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 31 13:03:51.950205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.950079 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 31 13:03:51.950205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.950100 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 31 13:03:51.950205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.950114 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 31 13:03:51.950205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.950174 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 31 13:03:51.950521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.950486 5532 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 31 13:03:51.955499 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.955476 5532 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 13:03:51.957626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.956915 5532 server.go:1263] "Started kubelet" Oct 31 13:03:51.957626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.956987 5532 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 31 13:03:51.957626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.957178 5532 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 13:03:51.957626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.957396 5532 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 31 13:03:51.957626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.957460 5532 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 31 13:03:51.957465 ip-10-0-128-41 systemd[1]: Started Kubernetes Kubelet. Oct 31 13:03:51.957981 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.957777 5532 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 31 13:03:51.958449 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.958431 5532 server.go:317] "Adding debug handlers to kubelet server" Oct 31 13:03:51.960683 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.960664 5532 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 31 13:03:51.960793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.960676 5532 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 31 13:03:51.960793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.960763 5532 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 31 13:03:51.960793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.960772 5532 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 31 13:03:51.960955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.960805 5532 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-30 12:57:39 +0000 UTC" deadline="2028-06-22 21:26:25.262087589 +0000 UTC" Oct 31 13:03:51.960955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.960825 5532 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14408h22m33.301265813s" Oct 31 13:03:51.960955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.960853 5532 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 31 13:03:51.966490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.966379 5532 factory.go:55] Registering systemd factory Oct 31 13:03:51.966490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.966403 5532 factory.go:223] Registration of the systemd container factory successfully Oct 31 13:03:51.966903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.966870 5532 factory.go:153] Registering CRI-O factory Oct 31 13:03:51.966903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.966890 5532 factory.go:223] Registration of the crio container factory successfully Oct 31 13:03:51.967025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.966952 5532 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 31 13:03:51.967025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.966980 5532 factory.go:103] Registering Raw factory Oct 31 13:03:51.967025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.966998 5532 manager.go:1199] Started watching for new ooms in manager Oct 31 13:03:51.967634 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967580 5532 manager.go:319] Starting recovery of all containers Oct 31 13:03:51.967725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967618 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9c948c-9fd3-41e3-8815-208cfb56fe33" volumeName="kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 31 13:03:51.967725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967673 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9c948c-9fd3-41e3-8815-208cfb56fe33" volumeName="kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-cluster-tls-config" seLinuxMountContext="" Oct 31 13:03:51.967831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967701 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccf935cb-694a-4d54-b7a3-f0be4bc1212c" volumeName="kubernetes.io/empty-dir/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-archives-path" seLinuxMountContext="" Oct 31 13:03:51.967831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967757 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2eab6ed-c191-4b42-899c-e816ab8d4a0b" volumeName="kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovnkube-script-lib" seLinuxMountContext="" Oct 31 13:03:51.967831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967788 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e507f475-28b0-43c1-86b1-5ee3e4c4dad4" volumeName="kubernetes.io/secret/e507f475-28b0-43c1-86b1-5ee3e4c4dad4-cert" seLinuxMountContext="" Oct 31 13:03:51.967831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967796 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0c07d8-f966-4acb-aea6-1ec8682b4877" volumeName="kubernetes.io/projected/ea0c07d8-f966-4acb-aea6-1ec8682b4877-kube-api-access-8bt8n" seLinuxMountContext="" Oct 31 13:03:51.967831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967804 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9c948c-9fd3-41e3-8815-208cfb56fe33" volumeName="kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-tls-assets" seLinuxMountContext="" Oct 31 13:03:51.967831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967816 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9c948c-9fd3-41e3-8815-208cfb56fe33" volumeName="kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 31 13:03:51.967831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967828 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9c948c-9fd3-41e3-8815-208cfb56fe33" volumeName="kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-volume" seLinuxMountContext="" Oct 31 13:03:51.968150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967915 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05baffb8-cba0-444a-989e-dc5df0c7bcd6" volumeName="kubernetes.io/empty-dir/05baffb8-cba0-444a-989e-dc5df0c7bcd6-tmp" seLinuxMountContext="" Oct 31 13:03:51.968150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967924 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9c948c-9fd3-41e3-8815-208cfb56fe33" volumeName="kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-metrics-client-ca" seLinuxMountContext="" Oct 31 13:03:51.968150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967936 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9c948c-9fd3-41e3-8815-208cfb56fe33" volumeName="kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 31 13:03:51.968150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.967960 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9c948c-9fd3-41e3-8815-208cfb56fe33" volumeName="kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-out" seLinuxMountContext="" Oct 31 13:03:51.968150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968013 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d38ca15-69a2-4950-9bf5-c77868a86cfb" volumeName="kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-service-ca" seLinuxMountContext="" Oct 31 13:03:51.968150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968037 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d38ca15-69a2-4950-9bf5-c77868a86cfb" volumeName="kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-oauth-serving-cert" seLinuxMountContext="" Oct 31 13:03:51.968150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968054 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="896e7579-dea0-49ee-92c5-cddf258c6905" volumeName="kubernetes.io/secret/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-tls" seLinuxMountContext="" Oct 31 13:03:51.968150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968064 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e25856e6-e50d-48d2-94a5-8966be78dca4" volumeName="kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-tls" seLinuxMountContext="" Oct 31 13:03:51.968150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968073 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19180e90-ffed-457b-9d2e-92645efe3491" volumeName="kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-cni-binary-copy" seLinuxMountContext="" Oct 31 13:03:51.968150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968084 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="25233e95-abc9-4f04-b7b2-65933229d471" volumeName="kubernetes.io/secret/25233e95-abc9-4f04-b7b2-65933229d471-tls-certificates" seLinuxMountContext="" Oct 31 13:03:51.968150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968097 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d38ca15-69a2-4950-9bf5-c77868a86cfb" volumeName="kubernetes.io/empty-dir/4d38ca15-69a2-4950-9bf5-c77868a86cfb-tmp" seLinuxMountContext="" Oct 31 13:03:51.968150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968108 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0b84e1d-390d-434d-b309-9440ec8195b6" volumeName="kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 31 13:03:51.968150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968117 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2eab6ed-c191-4b42-899c-e816ab8d4a0b" volumeName="kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovnkube-config" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968175 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc1ff283-237c-45f3-a90f-88cb75bcb863" volumeName="kubernetes.io/empty-dir/dc1ff283-237c-45f3-a90f-88cb75bcb863-tmp-dir" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968187 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0b84e1d-390d-434d-b309-9440ec8195b6" volumeName="kubernetes.io/projected/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-api-access-q88wd" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968196 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="018c5e83-597e-4c20-9063-5287b4c332a2" volumeName="kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968209 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9c948c-9fd3-41e3-8815-208cfb56fe33" volumeName="kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-web-config" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968218 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70ea50fe-3024-4a82-84a5-5c236706402b" volumeName="kubernetes.io/empty-dir/70ea50fe-3024-4a82-84a5-5c236706402b-tmp" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968227 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="896e7579-dea0-49ee-92c5-cddf258c6905" volumeName="kubernetes.io/empty-dir/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-textfile" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968254 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0b84e1d-390d-434d-b309-9440ec8195b6" volumeName="kubernetes.io/empty-dir/a0b84e1d-390d-434d-b309-9440ec8195b6-data-volume" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968265 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e507f475-28b0-43c1-86b1-5ee3e4c4dad4" volumeName="kubernetes.io/projected/e507f475-28b0-43c1-86b1-5ee3e4c4dad4-kube-api-access-4sfsw" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968273 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="896e7579-dea0-49ee-92c5-cddf258c6905" volumeName="kubernetes.io/configmap/896e7579-dea0-49ee-92c5-cddf258c6905-metrics-client-ca" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968282 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e3f5e50-ca1a-4a2a-af47-30a240b36bd5" volumeName="kubernetes.io/secret/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968293 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0104a2a9-918a-40bf-9ece-bfa7360a8d36" volumeName="kubernetes.io/projected/0104a2a9-918a-40bf-9ece-bfa7360a8d36-kube-api-access-jm6n8" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968301 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d38ca15-69a2-4950-9bf5-c77868a86cfb" volumeName="kubernetes.io/projected/4d38ca15-69a2-4950-9bf5-c77868a86cfb-kube-api-access-ml8wx" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968312 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5" volumeName="kubernetes.io/configmap/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-cni-binary-copy" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968429 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccf935cb-694a-4d54-b7a3-f0be4bc1212c" volumeName="kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-service-ca-bundle" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968632 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0c07d8-f966-4acb-aea6-1ec8682b4877" volumeName="kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls" seLinuxMountContext="" Oct 31 13:03:51.968718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968652 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0104a2a9-918a-40bf-9ece-bfa7360a8d36" volumeName="kubernetes.io/configmap/0104a2a9-918a-40bf-9ece-bfa7360a8d36-iptables-alerter-script" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968799 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54a25982-6507-405a-8e10-81d965ab7f4f" volumeName="kubernetes.io/configmap/54a25982-6507-405a-8e10-81d965ab7f4f-serviceca" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968814 5532 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968829 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05baffb8-cba0-444a-989e-dc5df0c7bcd6" volumeName="kubernetes.io/projected/05baffb8-cba0-444a-989e-dc5df0c7bcd6-kube-api-access-slrrr" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968856 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9c948c-9fd3-41e3-8815-208cfb56fe33" volumeName="kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968870 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d38ca15-69a2-4950-9bf5-c77868a86cfb" volumeName="kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-serving-cert" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968883 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1" volumeName="kubernetes.io/configmap/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1-konnectivity-ca" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968917 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1" volumeName="kubernetes.io/secret/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1-agent-certs" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968932 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5" volumeName="kubernetes.io/configmap/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-daemon-config" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968949 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9c948c-9fd3-41e3-8815-208cfb56fe33" volumeName="kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.968962 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="896e7579-dea0-49ee-92c5-cddf258c6905" volumeName="kubernetes.io/secret/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969027 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2eab6ed-c191-4b42-899c-e816ab8d4a0b" volumeName="kubernetes.io/secret/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovn-node-metrics-cert" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969045 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19180e90-ffed-457b-9d2e-92645efe3491" volumeName="kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-cni-sysctl-allowlist" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969059 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5" volumeName="kubernetes.io/projected/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-kube-api-access-sx88t" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969077 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="896e7579-dea0-49ee-92c5-cddf258c6905" volumeName="kubernetes.io/configmap/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 31 13:03:51.969395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969286 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e25856e6-e50d-48d2-94a5-8966be78dca4" volumeName="kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969416 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e25856e6-e50d-48d2-94a5-8966be78dca4" volumeName="kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969430 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1b612e1-018b-47d0-880a-cb879202e312" volumeName="kubernetes.io/projected/f1b612e1-018b-47d0-880a-cb879202e312-kube-api-access-5dk65" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969450 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d38ca15-69a2-4950-9bf5-c77868a86cfb" volumeName="kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-oauth-config" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969462 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70a6eabd-c790-46fa-a9f1-cff156624c2f" volumeName="kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969503 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c913d3bd-b2b9-42d0-a02b-e0474c15e14c" volumeName="kubernetes.io/projected/c913d3bd-b2b9-42d0-a02b-e0474c15e14c-kube-api-access-lkj5f" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969515 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9c948c-9fd3-41e3-8815-208cfb56fe33" volumeName="kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-main-db" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969550 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d38ca15-69a2-4950-9bf5-c77868a86cfb" volumeName="kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-config" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969562 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70a6eabd-c790-46fa-a9f1-cff156624c2f" volumeName="kubernetes.io/projected/70a6eabd-c790-46fa-a9f1-cff156624c2f-kube-api-access-cw7j7" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969583 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dc1ff283-237c-45f3-a90f-88cb75bcb863" volumeName="kubernetes.io/projected/dc1ff283-237c-45f3-a90f-88cb75bcb863-kube-api-access-nr7hd" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969626 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e25856e6-e50d-48d2-94a5-8966be78dca4" volumeName="kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969642 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0c07d8-f966-4acb-aea6-1ec8682b4877" volumeName="kubernetes.io/configmap/ea0c07d8-f966-4acb-aea6-1ec8682b4877-config-volume" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969659 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ea0c07d8-f966-4acb-aea6-1ec8682b4877" volumeName="kubernetes.io/empty-dir/ea0c07d8-f966-4acb-aea6-1ec8682b4877-tmp-dir" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969672 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05baffb8-cba0-444a-989e-dc5df0c7bcd6" volumeName="kubernetes.io/empty-dir/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-tuned" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969687 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2b9c948c-9fd3-41e3-8815-208cfb56fe33" volumeName="kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-kube-api-access-xdn2q" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969736 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="70ea50fe-3024-4a82-84a5-5c236706402b" volumeName="kubernetes.io/projected/70ea50fe-3024-4a82-84a5-5c236706402b-kube-api-access-7qj4l" seLinuxMountContext="" Oct 31 13:03:51.969964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.969762 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccf935cb-694a-4d54-b7a3-f0be4bc1212c" volumeName="kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-trusted-ca-bundle" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970137 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e25856e6-e50d-48d2-94a5-8966be78dca4" volumeName="kubernetes.io/configmap/e25856e6-e50d-48d2-94a5-8966be78dca4-metrics-client-ca" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970166 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e25856e6-e50d-48d2-94a5-8966be78dca4" volumeName="kubernetes.io/empty-dir/e25856e6-e50d-48d2-94a5-8966be78dca4-tmp" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970180 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19180e90-ffed-457b-9d2e-92645efe3491" volumeName="kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970193 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2aee367b-5b87-4bfe-9620-5adf75e25b63" volumeName="kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970206 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e3f5e50-ca1a-4a2a-af47-30a240b36bd5" volumeName="kubernetes.io/configmap/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-metrics-client-ca" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970217 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54a25982-6507-405a-8e10-81d965ab7f4f" volumeName="kubernetes.io/projected/54a25982-6507-405a-8e10-81d965ab7f4f-kube-api-access-nq4sr" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970229 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a0b84e1d-390d-434d-b309-9440ec8195b6" volumeName="kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970255 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2eab6ed-c191-4b42-899c-e816ab8d4a0b" volumeName="kubernetes.io/projected/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-kube-api-access-65x7x" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970266 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e25856e6-e50d-48d2-94a5-8966be78dca4" volumeName="kubernetes.io/projected/e25856e6-e50d-48d2-94a5-8966be78dca4-kube-api-access-k7mts" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970277 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e25856e6-e50d-48d2-94a5-8966be78dca4" volumeName="kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970288 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e3f5e50-ca1a-4a2a-af47-30a240b36bd5" volumeName="kubernetes.io/projected/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-kube-api-access-vxr4p" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970305 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3e3f5e50-ca1a-4a2a-af47-30a240b36bd5" volumeName="kubernetes.io/secret/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-openshift-state-metrics-tls" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970326 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="896e7579-dea0-49ee-92c5-cddf258c6905" volumeName="kubernetes.io/projected/896e7579-dea0-49ee-92c5-cddf258c6905-kube-api-access-tm6fc" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970338 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ccf935cb-694a-4d54-b7a3-f0be4bc1212c" volumeName="kubernetes.io/projected/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-kube-api-access-jfnsx" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970349 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d2eab6ed-c191-4b42-899c-e816ab8d4a0b" volumeName="kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-env-overrides" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970372 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e25856e6-e50d-48d2-94a5-8966be78dca4" volumeName="kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-grpc-tls" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970383 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19180e90-ffed-457b-9d2e-92645efe3491" volumeName="kubernetes.io/projected/19180e90-ffed-457b-9d2e-92645efe3491-kube-api-access-v79lg" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970394 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e45e4264-8ba9-4ec4-be49-ccc8d26472f1" volumeName="kubernetes.io/projected/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-kube-api-access-rg99f" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970407 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1b612e1-018b-47d0-880a-cb879202e312" volumeName="kubernetes.io/configmap/f1b612e1-018b-47d0-880a-cb879202e312-signing-cabundle" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970418 5532 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1b612e1-018b-47d0-880a-cb879202e312" volumeName="kubernetes.io/secret/f1b612e1-018b-47d0-880a-cb879202e312-signing-key" seLinuxMountContext="" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970428 5532 reconstruct.go:97] "Volume reconstruction finished" Oct 31 13:03:51.970643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.970436 5532 reconciler.go:29] "Reconciler: start to sync state" Oct 31 13:03:51.974710 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.974692 5532 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 31 13:03:51.992574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.992507 5532 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 31 13:03:51.994692 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.994670 5532 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 31 13:03:51.994831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.994819 5532 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 31 13:03:51.994919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.994910 5532 kubelet.go:2521] "Starting kubelet main sync loop" Oct 31 13:03:51.995029 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:03:51.995009 5532 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 31 13:03:51.999782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:51.999757 5532 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 31 13:03:52.038987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.038905 5532 generic.go:356] "Generic (PLEG): container finished" podID="896e7579-dea0-49ee-92c5-cddf258c6905" containerID="2b677e8a0d96833c3c46b8bec6f5ef89b7656fe6c548312cd60be33527fcd9a1" exitCode=0 Oct 31 13:03:52.055279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.055244 5532 generic.go:356] "Generic (PLEG): container finished" podID="37af5d4f145227f4335887a01dc2c4be" containerID="fd171edb3552762fd6f448c3835e83bb5820186d2707cc25302ee1fdec900077" exitCode=0 Oct 31 13:03:52.068468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.068437 5532 generic.go:356] "Generic (PLEG): container finished" podID="19180e90-ffed-457b-9d2e-92645efe3491" containerID="ab3aff25a31ee52e832322a93132b3b4bb25913ee1909a8f90b03872bed4fa01" exitCode=0 Oct 31 13:03:52.068468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.068465 5532 generic.go:356] "Generic (PLEG): container finished" podID="19180e90-ffed-457b-9d2e-92645efe3491" containerID="6131f117fa7d721279f822017d53dad5a9cb9233686ca6069e3a74465f0a11fd" exitCode=0 Oct 31 13:03:52.068658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.068477 5532 generic.go:356] "Generic (PLEG): container finished" podID="19180e90-ffed-457b-9d2e-92645efe3491" containerID="ab986b76768bf9f682c15b5d68938b4bfded14614a4cb257c6e1cf323033b377" exitCode=0 Oct 31 13:03:52.068658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.068487 5532 generic.go:356] "Generic (PLEG): container finished" podID="19180e90-ffed-457b-9d2e-92645efe3491" containerID="ee6374eb8ef727cd0ea9f58a0ff6e5f3ba6ff327cd51a7a6058afefafa4936d7" exitCode=0 Oct 31 13:03:52.068658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.068494 5532 generic.go:356] "Generic (PLEG): container finished" podID="19180e90-ffed-457b-9d2e-92645efe3491" containerID="61121f6a198dd263800eacd20102bf9aa75af91b49bf6f9f59ea3201e535b5a1" exitCode=0 Oct 31 13:03:52.068658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.068499 5532 generic.go:356] "Generic (PLEG): container finished" podID="19180e90-ffed-457b-9d2e-92645efe3491" containerID="050275a9c26b105282f2c96a45c577295c475e97337419ca3f74abbb25173396" exitCode=0 Oct 31 13:03:52.072672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.072612 5532 generic.go:356] "Generic (PLEG): container finished" podID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerID="5597b8699084216d1ebb3f8a5221731bf5b762337e5d25896e643d857b9a8089" exitCode=0 Oct 31 13:03:52.075849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.075831 5532 manager.go:324] Recovery completed Oct 31 13:03:52.095083 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:03:52.095052 5532 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 31 13:03:52.100815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.100789 5532 cpu_manager.go:226] "Starting" policy="none" Oct 31 13:03:52.100815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.100805 5532 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 31 13:03:52.100936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.100825 5532 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 31 13:03:52.100994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.100972 5532 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 31 13:03:52.101156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.100992 5532 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 31 13:03:52.101156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.101015 5532 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 31 13:03:52.101156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.101023 5532 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 31 13:03:52.101156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.101030 5532 policy_none.go:50] "Start" Oct 31 13:03:52.101156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.101040 5532 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 31 13:03:52.101156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.101052 5532 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 31 13:03:52.101420 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.101208 5532 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 31 13:03:52.101420 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.101219 5532 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 31 13:03:52.101420 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.101225 5532 policy_none.go:44] "Start" Oct 31 13:03:52.107649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.107100 5532 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 31 13:03:52.113795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.113776 5532 manager.go:341] "Starting Device Plugin manager" Oct 31 13:03:52.113892 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:03:52.113811 5532 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 31 13:03:52.113892 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.113823 5532 server.go:87] "Starting device plugin registration server" Oct 31 13:03:52.114130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.114115 5532 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 31 13:03:52.114207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.114132 5532 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 31 13:03:52.114361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.114343 5532 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 31 13:03:52.114454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.114423 5532 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 31 13:03:52.114454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.114451 5532 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 31 13:03:52.114523 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.114456 5532 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 31 13:03:52.114610 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.114523 5532 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T13:03:52.11444352Z","UUID":"bad27c9f-fd4a-4481-907b-fcc340dc5f73","Handler":null,"Name":"","Endpoint":""} Oct 31 13:03:52.116549 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:03:52.116527 5532 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 31 13:03:52.116701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.116637 5532 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 31 13:03:52.116701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.116658 5532 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 31 13:03:52.215099 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.215057 5532 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 31 13:03:52.217537 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.217514 5532 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 31 13:03:52.217676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.217548 5532 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 31 13:03:52.217676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.217561 5532 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-41.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 31 13:03:52.217676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.217631 5532 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.237866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.237836 5532 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.237992 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.237918 5532 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.295510 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.295365 5532 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal"] Oct 31 13:03:52.296423 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.296368 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" event={"ID":"37af5d4f145227f4335887a01dc2c4be","Type":"ContainerStarted","Data":"a911dcce01965c0acbfb6569bb22eb1645f6adb2b13f745f2765530f92f14bde"} Oct 31 13:03:52.296552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.296437 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" event={"ID":"37af5d4f145227f4335887a01dc2c4be","Type":"ContainerDied","Data":"fd171edb3552762fd6f448c3835e83bb5820186d2707cc25302ee1fdec900077"} Oct 31 13:03:52.296552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.296460 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" event={"ID":"37af5d4f145227f4335887a01dc2c4be","Type":"ContainerStarted","Data":"881b5c111023e271fe18bd5f38d10e6785b51aa743839d14c6155c1cef8a4fdb"} Oct 31 13:03:52.296552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.296509 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" event={"ID":"16434a8e882a2ee35299274e3b318497","Type":"ContainerStarted","Data":"965a2eb5235865b08b70aecb0a83643fdaca11c2141d0dd86f548b380973a52c"} Oct 31 13:03:52.296552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.296529 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" event={"ID":"16434a8e882a2ee35299274e3b318497","Type":"ContainerStarted","Data":"4516dd758e1729253b19122fc3125003c52f4727674a48c15031e664419aee42"} Oct 31 13:03:52.296854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.296831 5532 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.297332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.297315 5532 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.312704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.312679 5532 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 13:03:52.312833 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:03:52.312739 5532 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.313568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.313545 5532 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 31 13:03:52.313666 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:03:52.313589 5532 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.376863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.376833 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37af5d4f145227f4335887a01dc2c4be-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"37af5d4f145227f4335887a01dc2c4be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.376987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.376868 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37af5d4f145227f4335887a01dc2c4be-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"37af5d4f145227f4335887a01dc2c4be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.376987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.376887 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/16434a8e882a2ee35299274e3b318497-config\") pod \"kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"16434a8e882a2ee35299274e3b318497\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.477463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.477431 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/16434a8e882a2ee35299274e3b318497-config\") pod \"kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"16434a8e882a2ee35299274e3b318497\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.477463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.477464 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37af5d4f145227f4335887a01dc2c4be-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"37af5d4f145227f4335887a01dc2c4be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.477679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.477481 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37af5d4f145227f4335887a01dc2c4be-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"37af5d4f145227f4335887a01dc2c4be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.477679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.477528 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/37af5d4f145227f4335887a01dc2c4be-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"37af5d4f145227f4335887a01dc2c4be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.477679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.477543 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/16434a8e882a2ee35299274e3b318497-config\") pod \"kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"16434a8e882a2ee35299274e3b318497\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.477679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.477569 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/37af5d4f145227f4335887a01dc2c4be-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal\" (UID: \"37af5d4f145227f4335887a01dc2c4be\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal" Oct 31 13:03:52.932800 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.932762 5532 apiserver.go:52] "Watching apiserver" Oct 31 13:03:52.944080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.944058 5532 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 31 13:03:52.944697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.944676 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf","openshift-image-registry/node-ca-fbstt","openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2","openshift-console/console-64885574d7-l7vzc","openshift-insights/insights-runtime-extractor-nlmx2","kube-system/konnectivity-agent-rmt2g","openshift-console/downloads-6dd789c8b7-xc5m4","openshift-multus/multus-additional-cni-plugins-28gmd","openshift-network-diagnostics/network-check-target-7f7h9","openshift-ovn-kubernetes/ovnkube-node-26l6d","openshift-network-operator/iptables-alerter-zqcpp","openshift-service-ca/service-ca-5fffc97bdf-4ftlz","openshift-cluster-node-tuning-operator/tuned-qdllj","openshift-dns/node-resolver-l5vvf","kube-system/global-pull-secret-syncer-48bm7","openshift-insights/periodic-gathering-prddg-d22mn","openshift-multus/multus-ddf7z","openshift-monitoring/node-exporter-p5fj4","kube-system/kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4","openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj","openshift-multus/network-metrics-daemon-sxmc2","openshift-ingress-canary/ingress-canary-hgm5n","openshift-dns/dns-default-7rzwh","openshift-monitoring/alertmanager-main-0","openshift-monitoring/thanos-querier-57c987d54b-lc452","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-41.us-east-2.compute.internal","openshift-monitoring/openshift-state-metrics-7bd7489957-949gv"] Oct 31 13:03:52.950471 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.950445 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.950975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.950797 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.950975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.950912 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.950975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.950922 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 31 13:03:52.950975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.950928 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 31 13:03:52.950975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.950942 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.950975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.950929 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.950975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.950965 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.950975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.950971 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.951405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.951162 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-tmkp5\"" Oct 31 13:03:52.951405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.951184 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 31 13:03:52.951405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.951195 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 31 13:03:52.951405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.951218 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.951405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.951290 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-5rqjw\"" Oct 31 13:03:52.951405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.951309 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 31 13:03:52.951690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.951516 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-h72qs\"" Oct 31 13:03:52.951690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.951539 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 31 13:03:52.952430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.952407 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 31 13:03:52.952430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.952419 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-65v8h\"" Oct 31 13:03:52.952563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.952440 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 31 13:03:52.952563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.952447 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wr5k7\"" Oct 31 13:03:52.952563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.952475 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-s8mtw\"" Oct 31 13:03:52.952563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.952484 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-glfvs\"" Oct 31 13:03:52.953817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.953800 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 31 13:03:52.954040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.954026 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-s9d6g\"" Oct 31 13:03:52.954094 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.954063 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-j8xvf\"" Oct 31 13:03:52.954270 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.954259 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.954885 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.954867 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.955213 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.955192 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-r2wwf\"" Oct 31 13:03:52.955313 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.955227 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 31 13:03:52.955313 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.955298 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 31 13:03:52.955428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.955332 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nkxw2\"" Oct 31 13:03:52.955480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.955436 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 31 13:03:52.955778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.955763 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.955928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.955913 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.956040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956021 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.956040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956033 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.956190 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956041 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.956246 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956218 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 31 13:03:52.956640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956625 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 31 13:03:52.956640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956637 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 31 13:03:52.956774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956654 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-lxhpt\"" Oct 31 13:03:52.956774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956680 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.956774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956712 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-qbkfg\"" Oct 31 13:03:52.956774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956737 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 31 13:03:52.956958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956774 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 31 13:03:52.956958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956790 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-wdnw7\"" Oct 31 13:03:52.956958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956811 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.956958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.956837 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 31 13:03:52.957284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.957268 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-8dvth\"" Oct 31 13:03:52.957922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.957906 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.957981 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.957966 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 31 13:03:52.958034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.957991 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.958177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.958161 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tmfz7\"" Oct 31 13:03:52.958227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.958204 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.958316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.958305 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 31 13:03:52.958840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.958823 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 31 13:03:52.960375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.960349 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-b58t5\"" Oct 31 13:03:52.960456 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.960442 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 31 13:03:52.960749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.960729 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.961173 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.961153 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.961264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.961232 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 31 13:03:52.962340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.962321 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 31 13:03:52.962574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.962559 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 31 13:03:52.962783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.962768 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 31 13:03:52.962955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.962941 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 31 13:03:52.963138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.963125 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 31 13:03:52.963503 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.963477 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-66zbj\"" Oct 31 13:03:52.963581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.963508 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 31 13:03:52.963581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.963559 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 31 13:03:52.963581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.963570 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.963740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.963627 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lmr7q\"" Oct 31 13:03:52.963835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.963817 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 31 13:03:52.963887 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.963837 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.964059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.964036 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-zdl2b\"" Oct 31 13:03:52.964185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.964169 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 31 13:03:52.964328 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.964312 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-mk4zx\"" Oct 31 13:03:52.964386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.964375 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.964459 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.964439 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 31 13:03:52.964631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.962968 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 31 13:03:52.964717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.964700 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 31 13:03:52.964752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.964712 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-rrldl\"" Oct 31 13:03:52.965443 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.965401 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.965535 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.965440 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 31 13:03:52.965535 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.965448 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 31 13:03:52.965535 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.965508 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 31 13:03:52.965831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.965550 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-p4sq4\"" Oct 31 13:03:52.966188 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.966174 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2" Oct 31 13:03:52.969258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.969232 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 13:03:52.979783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.979762 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-node-log\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.979869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.979788 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-wtmp\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.979869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.979812 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr7hd\" (UniqueName: \"kubernetes.io/projected/dc1ff283-237c-45f3-a90f-88cb75bcb863-kube-api-access-nr7hd\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:03:52.979935 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.979873 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n54p8\" (UniqueName: \"kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8\") pod \"network-check-target-7f7h9\" (UID: \"018c5e83-597e-4c20-9063-5287b4c332a2\") " pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:52.980007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.979935 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65x7x\" (UniqueName: \"kubernetes.io/projected/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-kube-api-access-65x7x\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.980007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.979976 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-cnibin\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:52.980112 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980014 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-modprobe-d\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.980112 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980038 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-archives-path\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:52.980112 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980072 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d38ca15-69a2-4950-9bf5-c77868a86cfb-tmp\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:52.980240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980154 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d38ca15-69a2-4950-9bf5-c77868a86cfb-tmp\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:52.980240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980156 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-archives-path\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:52.980240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980182 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-slash\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.980240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980225 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq4sr\" (UniqueName: \"kubernetes.io/projected/54a25982-6507-405a-8e10-81d965ab7f4f-kube-api-access-nq4sr\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:03:52.980445 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980335 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.980445 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980362 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1ff283-237c-45f3-a90f-88cb75bcb863-tmp-dir\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:03:52.980445 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980436 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dc1ff283-237c-45f3-a90f-88cb75bcb863-tmp-dir\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:03:52.980545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980452 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-sys-fs\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:52.980545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980492 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg99f\" (UniqueName: \"kubernetes.io/projected/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-kube-api-access-rg99f\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:52.980545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980512 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f1b612e1-018b-47d0-880a-cb879202e312-signing-key\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:52.980545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980536 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:52.980736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980563 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e25856e6-e50d-48d2-94a5-8966be78dca4-metrics-client-ca\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:52.980736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980587 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.980736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980680 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:52.980736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980708 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:52.980919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980759 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-system-cni-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.980919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980770 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f1b612e1-018b-47d0-880a-cb879202e312-signing-key\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:52.980919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980780 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-kubelet\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.980919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980809 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:52.980919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980825 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0104a2a9-918a-40bf-9ece-bfa7360a8d36-iptables-alerter-script\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:03:52.980919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980836 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e25856e6-e50d-48d2-94a5-8966be78dca4-metrics-client-ca\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:52.980919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980851 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-cni-bin\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.980919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980876 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-systemd-units\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.980919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980879 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:52.980919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980912 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:52.981242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980970 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:52.981242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.980972 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysctl-d\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.981242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981018 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-hostroot\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.981242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981045 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0104a2a9-918a-40bf-9ece-bfa7360a8d36-iptables-alerter-script\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:03:52.981242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981043 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-cni-netd\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.981242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981080 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e507f475-28b0-43c1-86b1-5ee3e4c4dad4-cert\") pod \"ingress-canary-hgm5n\" (UID: \"e507f475-28b0-43c1-86b1-5ee3e4c4dad4\") " pod="openshift-ingress-canary/ingress-canary-hgm5n" Oct 31 13:03:52.981242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981101 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:52.981242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981217 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0104a2a9-918a-40bf-9ece-bfa7360a8d36-host-slash\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:03:52.981242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981237 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e507f475-28b0-43c1-86b1-5ee3e4c4dad4-cert\") pod \"ingress-canary-hgm5n\" (UID: \"e507f475-28b0-43c1-86b1-5ee3e4c4dad4\") " pod="openshift-ingress-canary/ingress-canary-hgm5n" Oct 31 13:03:52.981484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981255 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.981484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981287 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.981484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981333 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-device-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:52.981484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981364 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e25856e6-e50d-48d2-94a5-8966be78dca4-tmp\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:52.981484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981402 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e25856e6-e50d-48d2-94a5-8966be78dca4-tmp\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:52.981484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981420 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-log-socket\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.981484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981440 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-tuned\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.981484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981482 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-tuned\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.981820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981486 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-trusted-ca-bundle\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:52.981820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981506 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovn-node-metrics-cert\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.981820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981618 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:52.981820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981638 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfnsx\" (UniqueName: \"kubernetes.io/projected/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-kube-api-access-jfnsx\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:52.981820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981653 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.981820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981667 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0b84e1d-390d-434d-b309-9440ec8195b6-data-volume\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:52.981820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981684 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.981820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981703 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v79lg\" (UniqueName: \"kubernetes.io/projected/19180e90-ffed-457b-9d2e-92645efe3491-kube-api-access-v79lg\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:52.981820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981667 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovn-node-metrics-cert\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.981820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981725 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-sys\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.981820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981756 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-trusted-ca-bundle\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:52.981820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981785 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a0b84e1d-390d-434d-b309-9440ec8195b6-data-volume\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:52.981820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981806 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qj4l\" (UniqueName: \"kubernetes.io/projected/70ea50fe-3024-4a82-84a5-5c236706402b-kube-api-access-7qj4l\") pod \"downloads-6dd789c8b7-xc5m4\" (UID: \"70ea50fe-3024-4a82-84a5-5c236706402b\") " pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981848 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981886 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ea0c07d8-f966-4acb-aea6-1ec8682b4877-metrics-tls\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981882 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.981956 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdn2q\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-kube-api-access-xdn2q\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982055 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982107 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/896e7579-dea0-49ee-92c5-cddf258c6905-metrics-client-ca\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982125 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-lib-modules\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982139 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-var-lib-kubelet\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982200 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1-konnectivity-ca\") pod \"konnectivity-agent-rmt2g\" (UID: \"7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1\") " pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982234 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982257 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/896e7579-dea0-49ee-92c5-cddf258c6905-metrics-client-ca\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982261 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-etc-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982311 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkj5f\" (UniqueName: \"kubernetes.io/projected/c913d3bd-b2b9-42d0-a02b-e0474c15e14c-kube-api-access-lkj5f\") pod \"migrator-66df5d496b-lnxq4\" (UID: \"c913d3bd-b2b9-42d0-a02b-e0474c15e14c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982328 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54a25982-6507-405a-8e10-81d965ab7f4f-serviceca\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:03:52.982401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982385 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-cni-bin\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982433 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982480 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2aee367b-5b87-4bfe-9620-5adf75e25b63-kubelet-config\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982491 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1-konnectivity-ca\") pod \"konnectivity-agent-rmt2g\" (UID: \"7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1\") " pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982509 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysconfig\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982552 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25233e95-abc9-4f04-b7b2-65933229d471-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5vhhj\" (UID: \"25233e95-abc9-4f04-b7b2-65933229d471\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982583 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-oauth-config\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982633 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-config\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982661 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-cni-binary-copy\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982689 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54a25982-6507-405a-8e10-81d965ab7f4f-host\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982713 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-socket-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982741 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-oauth-config\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982750 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982784 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/25233e95-abc9-4f04-b7b2-65933229d471-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-5vhhj\" (UID: \"25233e95-abc9-4f04-b7b2-65933229d471\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982784 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-kubelet\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982834 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-env-overrides\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982847 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-config\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982868 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/19180e90-ffed-457b-9d2e-92645efe3491-cni-binary-copy\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982876 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982891 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54a25982-6507-405a-8e10-81d965ab7f4f-serviceca\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982942 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysctl-conf\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982961 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea0c07d8-f966-4acb-aea6-1ec8682b4877-tmp-dir\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982982 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ml8wx\" (UniqueName: \"kubernetes.io/projected/4d38ca15-69a2-4950-9bf5-c77868a86cfb-kube-api-access-ml8wx\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.982997 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxr4p\" (UniqueName: \"kubernetes.io/projected/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-kube-api-access-vxr4p\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983012 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-tls\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983016 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-env-overrides\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983028 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983072 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983086 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea0c07d8-f966-4acb-aea6-1ec8682b4877-tmp-dir\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983094 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sfsw\" (UniqueName: \"kubernetes.io/projected/e507f475-28b0-43c1-86b1-5ee3e4c4dad4-kube-api-access-4sfsw\") pod \"ingress-canary-hgm5n\" (UID: \"e507f475-28b0-43c1-86b1-5ee3e4c4dad4\") " pod="openshift-ingress-canary/ingress-canary-hgm5n" Oct 31 13:03:52.983080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983094 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-accelerators-collector-config\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983195 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-textfile\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983221 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-serving-cert\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983237 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983252 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-k8s-cni-cncf-io\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983273 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-daemon-config\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983275 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-textfile\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983270 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983307 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-tls\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983317 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0c07d8-f966-4acb-aea6-1ec8682b4877-config-volume\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983338 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983362 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-registration-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983380 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-cni-multus\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983380 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983421 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-serving-cert\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983413 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm6fc\" (UniqueName: \"kubernetes.io/projected/896e7579-dea0-49ee-92c5-cddf258c6905-kube-api-access-tm6fc\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983490 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-daemon-config\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983520 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-system-cni-dir\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983564 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-out\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983572 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ea0c07d8-f966-4acb-aea6-1ec8682b4877-config-volume\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983580 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q88wd\" (UniqueName: \"kubernetes.io/projected/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-api-access-q88wd\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983622 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-socket-dir-parent\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983656 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-out\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983694 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2aee367b-5b87-4bfe-9620-5adf75e25b63-dbus\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983714 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-host\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983731 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1-agent-certs\") pod \"konnectivity-agent-rmt2g\" (UID: \"7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1\") " pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983747 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-service-ca-bundle\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983784 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70ea50fe-3024-4a82-84a5-5c236706402b-tmp\") pod \"downloads-6dd789c8b7-xc5m4\" (UID: \"70ea50fe-3024-4a82-84a5-5c236706402b\") " pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983865 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/70ea50fe-3024-4a82-84a5-5c236706402b-tmp\") pod \"downloads-6dd789c8b7-xc5m4\" (UID: \"70ea50fe-3024-4a82-84a5-5c236706402b\") " pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983873 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-tls\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983907 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983961 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1-agent-certs\") pod \"konnectivity-agent-rmt2g\" (UID: \"7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1\") " pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.983988 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx88t\" (UniqueName: \"kubernetes.io/projected/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-kube-api-access-sx88t\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984020 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-run-netns\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984046 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984079 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-run\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984110 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-web-config\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984225 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-cni-binary-copy\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984257 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-ovn\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984232 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-service-ca-bundle\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984273 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-var-lib-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984309 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovnkube-config\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984341 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovnkube-script-lib\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984369 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-service-ca\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984387 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-cni-binary-copy\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984395 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm6n8\" (UniqueName: \"kubernetes.io/projected/0104a2a9-918a-40bf-9ece-bfa7360a8d36-kube-api-access-jm6n8\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984422 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.984489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984449 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984531 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovnkube-config\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984543 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-ovnkube-script-lib\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984559 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-conf-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984577 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-multus-certs\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984544 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-service-ca\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984623 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-systemd\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984654 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05baffb8-cba0-444a-989e-dc5df0c7bcd6-tmp\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984731 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/05baffb8-cba0-444a-989e-dc5df0c7bcd6-tmp\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984756 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984779 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984795 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984822 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-sys\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984841 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-oauth-serving-cert\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984862 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7mts\" (UniqueName: \"kubernetes.io/projected/e25856e6-e50d-48d2-94a5-8966be78dca4-kube-api-access-k7mts\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984879 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-cni-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984900 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slrrr\" (UniqueName: \"kubernetes.io/projected/05baffb8-cba0-444a-989e-dc5df0c7bcd6-kube-api-access-slrrr\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.984928 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0b84e1d-390d-434d-b309-9440ec8195b6-crio-socket\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985038 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-oauth-serving-cert\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985057 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985065 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/70a6eabd-c790-46fa-a9f1-cff156624c2f-metrics-certs\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985072 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f1b612e1-018b-47d0-880a-cb879202e312-signing-cabundle\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985119 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-netns\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985145 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-etc-kubernetes\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985178 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985191 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f1b612e1-018b-47d0-880a-cb879202e312-signing-cabundle\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985203 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-kubernetes\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985287 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985335 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985351 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985357 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985378 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bt8n\" (UniqueName: \"kubernetes.io/projected/ea0c07d8-f966-4acb-aea6-1ec8682b4877-kube-api-access-8bt8n\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985408 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cw7j7\" (UniqueName: \"kubernetes.io/projected/70a6eabd-c790-46fa-a9f1-cff156624c2f-kube-api-access-cw7j7\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985439 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dk65\" (UniqueName: \"kubernetes.io/projected/f1b612e1-018b-47d0-880a-cb879202e312-kube-api-access-5dk65\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985466 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-grpc-tls\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985491 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a0b84e1d-390d-434d-b309-9440ec8195b6-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985496 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-cnibin\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985522 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-root\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985559 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985579 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-systemd\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985705 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc1ff283-237c-45f3-a90f-88cb75bcb863-hosts-file\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:03:52.986169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.985846 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2aee367b-5b87-4bfe-9620-5adf75e25b63-original-pull-secret\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:52.991385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.991365 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr7hd\" (UniqueName: \"kubernetes.io/projected/dc1ff283-237c-45f3-a90f-88cb75bcb863-kube-api-access-nr7hd\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:03:52.991450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.991435 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65x7x\" (UniqueName: \"kubernetes.io/projected/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-kube-api-access-65x7x\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:52.991486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.991447 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq4sr\" (UniqueName: \"kubernetes.io/projected/54a25982-6507-405a-8e10-81d965ab7f4f-kube-api-access-nq4sr\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:03:52.991571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.991552 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n54p8\" (UniqueName: \"kubernetes.io/projected/018c5e83-597e-4c20-9063-5287b4c332a2-kube-api-access-n54p8\") pod \"network-check-target-7f7h9\" (UID: \"018c5e83-597e-4c20-9063-5287b4c332a2\") " pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:52.991629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.991614 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg99f\" (UniqueName: \"kubernetes.io/projected/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-kube-api-access-rg99f\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:52.992735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.992715 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qj4l\" (UniqueName: \"kubernetes.io/projected/70ea50fe-3024-4a82-84a5-5c236706402b-kube-api-access-7qj4l\") pod \"downloads-6dd789c8b7-xc5m4\" (UID: \"70ea50fe-3024-4a82-84a5-5c236706402b\") " pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:03:52.992806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.992757 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkj5f\" (UniqueName: \"kubernetes.io/projected/c913d3bd-b2b9-42d0-a02b-e0474c15e14c-kube-api-access-lkj5f\") pod \"migrator-66df5d496b-lnxq4\" (UID: \"c913d3bd-b2b9-42d0-a02b-e0474c15e14c\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lnxq4" Oct 31 13:03:52.992806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.992785 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdn2q\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-kube-api-access-xdn2q\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:52.993606 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.993578 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v79lg\" (UniqueName: \"kubernetes.io/projected/19180e90-ffed-457b-9d2e-92645efe3491-kube-api-access-v79lg\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:52.993649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.993582 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfnsx\" (UniqueName: \"kubernetes.io/projected/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-kube-api-access-jfnsx\") pod \"periodic-gathering-prddg-d22mn\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:03:52.999459 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.999355 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q88wd\" (UniqueName: \"kubernetes.io/projected/a0b84e1d-390d-434d-b309-9440ec8195b6-kube-api-access-q88wd\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:52.999869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.999557 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm6n8\" (UniqueName: \"kubernetes.io/projected/0104a2a9-918a-40bf-9ece-bfa7360a8d36-kube-api-access-jm6n8\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:03:52.999869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.999837 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sfsw\" (UniqueName: \"kubernetes.io/projected/e507f475-28b0-43c1-86b1-5ee3e4c4dad4-kube-api-access-4sfsw\") pod \"ingress-canary-hgm5n\" (UID: \"e507f475-28b0-43c1-86b1-5ee3e4c4dad4\") " pod="openshift-ingress-canary/ingress-canary-hgm5n" Oct 31 13:03:52.999869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.999848 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slrrr\" (UniqueName: \"kubernetes.io/projected/05baffb8-cba0-444a-989e-dc5df0c7bcd6-kube-api-access-slrrr\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.000020 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:52.999910 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm6fc\" (UniqueName: \"kubernetes.io/projected/896e7579-dea0-49ee-92c5-cddf258c6905-kube-api-access-tm6fc\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:53.000346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.000330 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx88t\" (UniqueName: \"kubernetes.io/projected/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-kube-api-access-sx88t\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.000429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.000409 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cw7j7\" (UniqueName: \"kubernetes.io/projected/70a6eabd-c790-46fa-a9f1-cff156624c2f-kube-api-access-cw7j7\") pod \"network-metrics-daemon-sxmc2\" (UID: \"70a6eabd-c790-46fa-a9f1-cff156624c2f\") " pod="openshift-multus/network-metrics-daemon-sxmc2" Oct 31 13:03:53.000512 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.000491 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxr4p\" (UniqueName: \"kubernetes.io/projected/3e3f5e50-ca1a-4a2a-af47-30a240b36bd5-kube-api-access-vxr4p\") pod \"openshift-state-metrics-7bd7489957-949gv\" (UID: \"3e3f5e50-ca1a-4a2a-af47-30a240b36bd5\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-949gv" Oct 31 13:03:53.000675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.000661 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ml8wx\" (UniqueName: \"kubernetes.io/projected/4d38ca15-69a2-4950-9bf5-c77868a86cfb-kube-api-access-ml8wx\") pod \"console-64885574d7-l7vzc\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:53.004224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.004208 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dk65\" (UniqueName: \"kubernetes.io/projected/f1b612e1-018b-47d0-880a-cb879202e312-kube-api-access-5dk65\") pod \"service-ca-5fffc97bdf-4ftlz\" (UID: \"f1b612e1-018b-47d0-880a-cb879202e312\") " pod="openshift-service-ca/service-ca-5fffc97bdf-4ftlz" Oct 31 13:03:53.004864 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.004849 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bt8n\" (UniqueName: \"kubernetes.io/projected/ea0c07d8-f966-4acb-aea6-1ec8682b4877-kube-api-access-8bt8n\") pod \"dns-default-7rzwh\" (UID: \"ea0c07d8-f966-4acb-aea6-1ec8682b4877\") " pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:53.004920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.004910 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7mts\" (UniqueName: \"kubernetes.io/projected/e25856e6-e50d-48d2-94a5-8966be78dca4-kube-api-access-k7mts\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:53.050392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.050371 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 31 13:03:53.050552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.050532 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 31 13:03:53.050655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.050638 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 31 13:03:53.050929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.050912 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:53.051054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051039 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 13:03:53.051114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051039 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 13:03:53.051114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051065 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 31 13:03:53.051114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051077 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-25mffia9fn4bj\"" Oct 31 13:03:53.051114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051104 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 13:03:53.051114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051109 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 13:03:53.051338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051105 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 13:03:53.051338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051125 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 31 13:03:53.051338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051099 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 13:03:53.051338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051151 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-qj4t9\"" Oct 31 13:03:53.051338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051179 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 13:03:53.051338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051253 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nztnk\"" Oct 31 13:03:53.051338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051331 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 13:03:53.051640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051406 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qx7z5\"" Oct 31 13:03:53.051640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051497 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 31 13:03:53.051640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.051532 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:53.053208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.053183 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:53.054316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.054290 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:53.054434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.054312 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:53.054434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.054343 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-tls\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:53.054527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.054457 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-web-config\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:53.055676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.055643 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:53.055676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.055658 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:53.055779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.055731 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:53.055843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.055815 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-grpc-tls\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:53.060785 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.060761 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:53.060888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.060865 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-volume\") pod \"alertmanager-main-0\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:03:53.060956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.060928 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/e25856e6-e50d-48d2-94a5-8966be78dca4-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57c987d54b-lc452\" (UID: \"e25856e6-e50d-48d2-94a5-8966be78dca4\") " pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:53.061158 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.061142 5532 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 31 13:03:53.085975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.085948 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-sys-fs\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:53.086093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.085981 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-system-cni-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.086093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.085995 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-kubelet\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.086093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086014 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/849c2ed1-3d50-4936-a2a9-e734e01108e1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wzvz2\" (UID: \"849c2ed1-3d50-4936-a2a9-e734e01108e1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2" Oct 31 13:03:53.086093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086031 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-cni-bin\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.086093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086055 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-systemd-units\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.086093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086082 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-system-cni-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086085 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-kubelet\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086113 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysctl-d\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086138 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-hostroot\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086143 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-systemd-units\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086160 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-cni-netd\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086182 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0104a2a9-918a-40bf-9ece-bfa7360a8d36-host-slash\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086190 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-hostroot\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086203 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086206 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-sys-fs\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086215 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-cni-netd\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086236 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-run-ovn-kubernetes\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086243 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-device-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086257 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0104a2a9-918a-40bf-9ece-bfa7360a8d36-host-slash\") pod \"iptables-alerter-zqcpp\" (UID: \"0104a2a9-918a-40bf-9ece-bfa7360a8d36\") " pod="openshift-network-operator/iptables-alerter-zqcpp" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086242 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-cni-bin\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086274 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-log-socket\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086307 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086325 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-sys\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086307 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-device-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086309 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-log-socket\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086345 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-lib-modules\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086349 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086271 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysctl-d\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086382 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-var-lib-kubelet\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086377 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-sys\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086409 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:53.086425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086434 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-etc-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086449 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-var-lib-kubelet\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086454 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-lib-modules\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086459 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-cni-bin\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086485 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-etc-selinux\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086490 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086497 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-cni-bin\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086493 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-etc-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086517 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2aee367b-5b87-4bfe-9620-5adf75e25b63-kubelet-config\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086537 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysconfig\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086554 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54a25982-6507-405a-8e10-81d965ab7f4f-host\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086521 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086577 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-socket-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086621 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-kubelet\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086628 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2aee367b-5b87-4bfe-9620-5adf75e25b63-kubelet-config\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086639 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysconfig\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086643 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54a25982-6507-405a-8e10-81d965ab7f4f-host\") pod \"node-ca-fbstt\" (UID: \"54a25982-6507-405a-8e10-81d965ab7f4f\") " pod="openshift-image-registry/node-ca-fbstt" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086645 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysctl-conf\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086673 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-kubelet\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086681 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086704 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-k8s-cni-cncf-io\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086702 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-socket-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086724 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-registration-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086749 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-cni-multus\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086772 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-system-cni-dir\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086777 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-sysctl-conf\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086766 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-registration-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086796 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e45e4264-8ba9-4ec4-be49-ccc8d26472f1-kubelet-dir\") pod \"aws-ebs-csi-driver-node-v4sjf\" (UID: \"e45e4264-8ba9-4ec4-be49-ccc8d26472f1\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-v4sjf" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086811 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-k8s-cni-cncf-io\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086819 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-var-lib-cni-multus\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086823 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-system-cni-dir\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086833 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-socket-dir-parent\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086849 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2aee367b-5b87-4bfe-9620-5adf75e25b63-dbus\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086862 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-host\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086882 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-socket-dir-parent\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086888 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-run-netns\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086913 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086919 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2aee367b-5b87-4bfe-9620-5adf75e25b63-dbus\") pod \"global-pull-secret-syncer-48bm7\" (UID: \"2aee367b-5b87-4bfe-9620-5adf75e25b63\") " pod="kube-system/global-pull-secret-syncer-48bm7" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086923 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-host\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086939 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-run\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086964 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-ovn\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086966 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-run-netns\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086978 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-var-lib-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.086995 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-conf-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087002 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-tuning-conf-dir\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087009 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-ovn\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087013 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-multus-certs\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087010 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-run\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087043 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-conf-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087051 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-var-lib-openvswitch\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087056 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-multus-certs\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087077 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-systemd\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087095 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-sys\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087112 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-run-systemd\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087115 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-cni-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087130 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0b84e1d-390d-434d-b309-9440ec8195b6-crio-socket\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087144 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-netns\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087158 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-etc-kubernetes\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087176 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-kubernetes\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087188 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-multus-cni-dir\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087202 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4wg7\" (UniqueName: \"kubernetes.io/projected/849c2ed1-3d50-4936-a2a9-e734e01108e1-kube-api-access-x4wg7\") pod \"monitoring-plugin-77f4f76594-wzvz2\" (UID: \"849c2ed1-3d50-4936-a2a9-e734e01108e1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087205 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-sys\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087221 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-host-run-netns\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087222 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a0b84e1d-390d-434d-b309-9440ec8195b6-crio-socket\") pod \"insights-runtime-extractor-nlmx2\" (UID: \"a0b84e1d-390d-434d-b309-9440ec8195b6\") " pod="openshift-insights/insights-runtime-extractor-nlmx2" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087234 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-etc-kubernetes\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087237 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-kubernetes\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087247 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-cnibin\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087271 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-root\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087287 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-systemd\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087309 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc1ff283-237c-45f3-a90f-88cb75bcb863-hosts-file\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087323 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-node-log\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087327 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-root\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087336 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-wtmp\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087373 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-systemd\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087383 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/896e7579-dea0-49ee-92c5-cddf258c6905-node-exporter-wtmp\") pod \"node-exporter-p5fj4\" (UID: \"896e7579-dea0-49ee-92c5-cddf258c6905\") " pod="openshift-monitoring/node-exporter-p5fj4" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087400 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/dc1ff283-237c-45f3-a90f-88cb75bcb863-hosts-file\") pod \"node-resolver-l5vvf\" (UID: \"dc1ff283-237c-45f3-a90f-88cb75bcb863\") " pod="openshift-dns/node-resolver-l5vvf" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087407 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-node-log\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087423 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5-cnibin\") pod \"multus-ddf7z\" (UID: \"81aa31ec-a7a8-4ce8-9097-1c3cec6b49c5\") " pod="openshift-multus/multus-ddf7z" Oct 31 13:03:53.087407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087449 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-cnibin\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:53.089473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087464 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-modprobe-d\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.089473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087485 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-slash\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.089473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087504 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/19180e90-ffed-457b-9d2e-92645efe3491-cnibin\") pod \"multus-additional-cni-plugins-28gmd\" (UID: \"19180e90-ffed-457b-9d2e-92645efe3491\") " pod="openshift-multus/multus-additional-cni-plugins-28gmd" Oct 31 13:03:53.089473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087537 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d2eab6ed-c191-4b42-899c-e816ab8d4a0b-host-slash\") pod \"ovnkube-node-26l6d\" (UID: \"d2eab6ed-c191-4b42-899c-e816ab8d4a0b\") " pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.089473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.087578 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/05baffb8-cba0-444a-989e-dc5df0c7bcd6-etc-modprobe-d\") pod \"tuned-qdllj\" (UID: \"05baffb8-cba0-444a-989e-dc5df0c7bcd6\") " pod="openshift-cluster-node-tuning-operator/tuned-qdllj" Oct 31 13:03:53.187928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.187834 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/849c2ed1-3d50-4936-a2a9-e734e01108e1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wzvz2\" (UID: \"849c2ed1-3d50-4936-a2a9-e734e01108e1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2" Oct 31 13:03:53.188098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.187930 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4wg7\" (UniqueName: \"kubernetes.io/projected/849c2ed1-3d50-4936-a2a9-e734e01108e1-kube-api-access-x4wg7\") pod \"monitoring-plugin-77f4f76594-wzvz2\" (UID: \"849c2ed1-3d50-4936-a2a9-e734e01108e1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2" Oct 31 13:03:53.188289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.188264 5532 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 31 13:03:53.190614 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.190578 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/849c2ed1-3d50-4936-a2a9-e734e01108e1-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-wzvz2\" (UID: \"849c2ed1-3d50-4936-a2a9-e734e01108e1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2" Oct 31 13:03:53.200660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.200623 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4wg7\" (UniqueName: \"kubernetes.io/projected/849c2ed1-3d50-4936-a2a9-e734e01108e1-kube-api-access-x4wg7\") pod \"monitoring-plugin-77f4f76594-wzvz2\" (UID: \"849c2ed1-3d50-4936-a2a9-e734e01108e1\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2" Oct 31 13:03:53.258287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.258268 5532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:03:53.278432 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.278411 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2" Oct 31 13:03:53.551740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.551658 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2"] Oct 31 13:03:53.555296 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:53.555258 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849c2ed1_3d50_4936_a2a9_e734e01108e1.slice/crio-227a6cd9b292b4c0c54304dcb3f4bb56f3ddee0d84cddd74bae12a7a092ec2d4 WatchSource:0}: Error finding container 227a6cd9b292b4c0c54304dcb3f4bb56f3ddee0d84cddd74bae12a7a092ec2d4: Status 404 returned error can't find the container with id 227a6cd9b292b4c0c54304dcb3f4bb56f3ddee0d84cddd74bae12a7a092ec2d4 Oct 31 13:03:53.616204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.616179 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:03:53.772965 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.772934 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:03:53.791932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.791906 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.797576 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.797546 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:03:53.797895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.797855 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 13:03:53.798005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.797952 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 13:03:53.798005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.797957 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 13:03:53.798261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.798056 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 13:03:53.798261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.798066 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 13:03:53.798261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.798122 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 13:03:53.798261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.798138 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 13:03:53.798261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.798184 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 13:03:53.798261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.798190 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 13:03:53.798261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.798253 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 13:03:53.798571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.798256 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a7uov9imvc59d\"" Oct 31 13:03:53.798571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.798367 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 13:03:53.799479 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.799461 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-896z6\"" Oct 31 13:03:53.799580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.799509 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 13:03:53.800157 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.800136 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 13:03:53.807917 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.807577 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 13:03:53.807917 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.807667 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 13:03:53.893131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893090 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893137 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893230 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893267 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-config-out\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893294 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893385 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893433 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893608 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893529 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893608 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893557 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893587 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893659 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dcwc\" (UniqueName: \"kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-kube-api-access-5dcwc\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893678 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893703 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893717 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893735 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-web-config\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.893875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893790 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.894002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.893974 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.894072 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.894047 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-config\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.894182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.894165 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.894230 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.894195 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995025 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995075 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-config\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995108 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995134 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995163 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995186 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995218 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995238 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-config-out\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995261 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995280 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995310 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995327 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995345 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995364 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995382 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dcwc\" (UniqueName: \"kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-kube-api-access-5dcwc\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995398 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995424 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995439 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995456 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-web-config\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.995669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995471 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.996193 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.995763 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.996193 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.996143 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.996690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.996655 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.997144 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.997119 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.997585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.997556 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:53.999877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:53.999831 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-config-out\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.003386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.003286 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.003386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.001836 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.003386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.002213 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-config\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.003386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.002433 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.003386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.002799 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.003386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.002955 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.003763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.000706 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.007276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.005573 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.007276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.005715 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.007276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.006175 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.007276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.006195 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-web-config\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.007519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.007307 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.008361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.008198 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.009774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.009752 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dcwc\" (UniqueName: \"kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-kube-api-access-5dcwc\") pod \"prometheus-k8s-0\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.089666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.089637 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" event={"ID":"e25856e6-e50d-48d2-94a5-8966be78dca4","Type":"ContainerStarted","Data":"5848e14f2530934411df7b727a2e20aea69495c5091c0ebf3972c7e07976c131"} Oct 31 13:03:54.093752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.093703 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2" event={"ID":"849c2ed1-3d50-4936-a2a9-e734e01108e1","Type":"ContainerStarted","Data":"227a6cd9b292b4c0c54304dcb3f4bb56f3ddee0d84cddd74bae12a7a092ec2d4"} Oct 31 13:03:54.110348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.110324 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:03:54.237036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.236824 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:03:54.333319 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.333110 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bcd5bb579-ngn5l"] Oct 31 13:03:54.350810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.350713 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bcd5bb579-ngn5l"] Oct 31 13:03:54.351197 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.351119 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.399908 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.399875 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntqfw\" (UniqueName: \"kubernetes.io/projected/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-kube-api-access-ntqfw\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.400087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.399920 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-config\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.400087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.399944 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-oauth-config\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.400087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.399960 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-oauth-serving-cert\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.400087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.400007 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-service-ca\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.400087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.400030 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-tmp\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.400087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.400054 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-trusted-ca-bundle\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.400372 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.400099 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-serving-cert\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.416613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.416561 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.501913 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-service-ca\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.502003 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-tmp\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.502036 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-trusted-ca-bundle\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.502094 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-serving-cert\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.502166 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntqfw\" (UniqueName: \"kubernetes.io/projected/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-kube-api-access-ntqfw\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.502207 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-config\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.502240 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-oauth-config\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.502272 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-oauth-serving-cert\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.504196 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-trusted-ca-bundle\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.504561 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-service-ca\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.504907 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-tmp\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.504926 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-config\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.505493 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-oauth-serving-cert\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.508829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.506485 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-serving-cert\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.518139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.518111 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-oauth-config\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.518282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.518167 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntqfw\" (UniqueName: \"kubernetes.io/projected/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-kube-api-access-ntqfw\") pod \"console-6bcd5bb579-ngn5l\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.574559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.574444 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:03:54.578651 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:54.578586 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50430d1_c382_4a45_9672_d4cb509f8aaf.slice/crio-11f17d2cc472af966598a715bd1c5677343800f4fab9d87543014b51364bf583 WatchSource:0}: Error finding container 11f17d2cc472af966598a715bd1c5677343800f4fab9d87543014b51364bf583: Status 404 returned error can't find the container with id 11f17d2cc472af966598a715bd1c5677343800f4fab9d87543014b51364bf583 Oct 31 13:03:54.676081 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.676046 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:54.812224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.812194 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7rzwh" Oct 31 13:03:54.976167 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:54.976133 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bcd5bb579-ngn5l"] Oct 31 13:03:54.980130 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:03:54.980095 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bfa1c3d_3bb4_42b6_b42f_ac83ef685ddb.slice/crio-e91821635399a920ae255ae47410ad66d792e1c7e1bd5a899ba94dbae9e6b10a WatchSource:0}: Error finding container e91821635399a920ae255ae47410ad66d792e1c7e1bd5a899ba94dbae9e6b10a: Status 404 returned error can't find the container with id e91821635399a920ae255ae47410ad66d792e1c7e1bd5a899ba94dbae9e6b10a Oct 31 13:03:55.069493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:55.068718 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" Oct 31 13:03:55.074008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:55.073987 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-5vhhj" Oct 31 13:03:55.108544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:55.107826 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcd5bb579-ngn5l" event={"ID":"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb","Type":"ContainerStarted","Data":"617f5c686619d1dc97780f2b1a717eae1fe721ba382a8cc3a8dbb5d0743faf90"} Oct 31 13:03:55.108544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:55.107872 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcd5bb579-ngn5l" event={"ID":"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb","Type":"ContainerStarted","Data":"e91821635399a920ae255ae47410ad66d792e1c7e1bd5a899ba94dbae9e6b10a"} Oct 31 13:03:55.114742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:55.114709 5532 generic.go:356] "Generic (PLEG): container finished" podID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerID="83ed47794c080bd2a987bdfc04eda6dfc1f6304c1af1a46b237ea2e03031c068" exitCode=0 Oct 31 13:03:55.114875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:55.114813 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerDied","Data":"83ed47794c080bd2a987bdfc04eda6dfc1f6304c1af1a46b237ea2e03031c068"} Oct 31 13:03:55.114936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:55.114882 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerStarted","Data":"11f17d2cc472af966598a715bd1c5677343800f4fab9d87543014b51364bf583"} Oct 31 13:03:55.125289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:55.125021 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" event={"ID":"e25856e6-e50d-48d2-94a5-8966be78dca4","Type":"ContainerStarted","Data":"37a32e2428f40f18604e55fa2fc4f49ca57e13d1313a7905a94eee5664f56cac"} Oct 31 13:03:55.125289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:55.125078 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" event={"ID":"e25856e6-e50d-48d2-94a5-8966be78dca4","Type":"ContainerStarted","Data":"e3cd219bb81ccb501724dfc80c7eb8772c5218f1e33ceed0334ba1675362e0ec"} Oct 31 13:03:55.136461 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:55.135740 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6bcd5bb579-ngn5l" podStartSLOduration=1.135722125 podStartE2EDuration="1.135722125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:55.135402803 +0000 UTC m=+3.281146632" watchObservedRunningTime="2026-10-31 13:03:55.135722125 +0000 UTC m=+3.281465954" Oct 31 13:03:56.135310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:56.135232 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerStarted","Data":"393beb865019774cebee9b24cb3e5da3673356faab44321dd6b846c68887f024"} Oct 31 13:03:56.135702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:56.135310 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerStarted","Data":"05da427806f0d54a530a3fbc4beca775c405034584b8604aa329939de1a59eb4"} Oct 31 13:03:56.135702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:56.135338 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerStarted","Data":"2100d067125a7c00cf540a5f2a668f44a2008b6afa0c8f8e2186f48c1c434a14"} Oct 31 13:03:56.135702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:56.135362 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerStarted","Data":"334adab89868ffd6c4c096259d08fac33f717760223dd42e143421cddfcff1ee"} Oct 31 13:03:56.135702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:56.135395 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerStarted","Data":"fd46c35c975ae610e101333d0f86341a6f40c20f4986927bc914ff43f114d910"} Oct 31 13:03:56.135702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:56.135420 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerStarted","Data":"b374bf9aee74f40be27dd6864f78393762d9259c76f1cf7026c927f765535c44"} Oct 31 13:03:56.138827 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:56.138792 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2" event={"ID":"849c2ed1-3d50-4936-a2a9-e734e01108e1","Type":"ContainerStarted","Data":"88dcd6825d83ec86f4f9d35aaca67213f7f2bbc46df9668570eda9ed5e7e03b8"} Oct 31 13:03:56.145369 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:56.145337 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" event={"ID":"e25856e6-e50d-48d2-94a5-8966be78dca4","Type":"ContainerStarted","Data":"17c9f2bc96934fc4ad0287d28cadad14ebfd7f1ff6f880deb8232d376ac31d33"} Oct 31 13:03:56.145501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:56.145406 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" event={"ID":"e25856e6-e50d-48d2-94a5-8966be78dca4","Type":"ContainerStarted","Data":"d080252f0f40c50de17cfca9d3f7bec0bb461d7e582af797c742b8d64405685d"} Oct 31 13:03:56.145501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:56.145446 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" event={"ID":"e25856e6-e50d-48d2-94a5-8966be78dca4","Type":"ContainerStarted","Data":"9d6e2be166bd1ecb618dd59ce055647ce8427bcfdea354dd919ac1d96b9e04ca"} Oct 31 13:03:56.228081 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:56.228047 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:03:56.229553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:56.229497 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2" podStartSLOduration=2.308872236 podStartE2EDuration="4.22947958s" totalImagesPullingTime="1.920607344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:56.228561696 +0000 UTC m=+4.374305524" watchObservedRunningTime="2026-10-31 13:03:56.22947958 +0000 UTC m=+4.375223407" Oct 31 13:03:56.419691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:56.419655 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:03:57.152289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:57.152254 5532 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 13:03:57.792017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:57.791983 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:03:58.233702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:58.233666 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2" Oct 31 13:03:58.234137 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:58.233784 5532 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 13:03:58.240485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:58.240453 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-wzvz2" Oct 31 13:03:59.165715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:59.165680 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerStarted","Data":"487b4a83f2f5f401fbcf9521d948de749d34abe579934698633a22edd420c616"} Oct 31 13:03:59.165873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:59.165763 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerStarted","Data":"d10cae6eda6aa669c0188cb8e44f3f26c310801ef231d630a93a70b85266459e"} Oct 31 13:03:59.165873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:59.165809 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerStarted","Data":"5efcf5d9875c1ee6550c100c7b743286e4f846023aefde0ad98f5a608c41c9d8"} Oct 31 13:03:59.165873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:59.165849 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerStarted","Data":"d321d37eef3a14597902e72643ba03d971884338fc517669c654ec9f3af79861"} Oct 31 13:03:59.166088 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:59.165890 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerStarted","Data":"85ebe3b76ef5a1a9cc3d0bff52702bbc61abf879559cef44ad0698322963ad3d"} Oct 31 13:03:59.166088 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:59.165921 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerStarted","Data":"1eb0a60c3ac6bafee09671af8bd29e30ba5e3fbed611e5a74ca08e637c02a3b5"} Oct 31 13:03:59.205893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:03:59.205800 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.624869521 podStartE2EDuration="6.205776247s" totalImagesPullingTime="3.557185911s" totalInitContainerRuntime="23.720815ms" isStatefulPod=false podCreationTimestamp="2026-10-31 13:03:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:03:59.204447384 +0000 UTC m=+7.350191212" watchObservedRunningTime="2026-10-31 13:03:59.205776247 +0000 UTC m=+7.351520072" Oct 31 13:04:01.542499 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:01.542413 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:04:01.548361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:01.548333 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:04:01.596870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:01.596827 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:04:01.765949 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:04:01.765912 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:04:02.189994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:02.189966 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:04:02.245163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:02.245132 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64885574d7-l7vzc"] Oct 31 13:04:02.642622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:02.642524 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:04:02.643072 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:02.642791 5532 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 31 13:04:02.779703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:02.779672 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:04:03.162003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:03.161975 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57c987d54b-lc452" Oct 31 13:04:05.711063 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:05.711034 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rmt2g" Oct 31 13:04:10.048961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:10.048931 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-26l6d" Oct 31 13:04:11.224337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:11.224294 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-xc5m4" event={"ID":"70ea50fe-3024-4a82-84a5-5c236706402b","Type":"ContainerStarted","Data":"156bb86d1b5a16842f46ed42fa61551eff87ae0dd256810f3d133d01008abf6b"} Oct 31 13:04:11.224958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:11.224733 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:04:11.242067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:11.242031 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-xc5m4" Oct 31 13:04:24.113063 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:24.113031 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-7f7h9" Oct 31 13:04:27.268892 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.268847 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-64885574d7-l7vzc" podUID="4d38ca15-69a2-4950-9bf5-c77868a86cfb" containerName="console" containerID="cri-o://445f68b339fbb070e4bb7fa7195c1255bdc1bdc6ad63a526713ed16ff6c25231" gracePeriod=15 Oct 31 13:04:27.554526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.554468 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:04:27.610189 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.610166 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-oauth-config\") pod \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " Oct 31 13:04:27.610345 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.610200 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d38ca15-69a2-4950-9bf5-c77868a86cfb-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d38ca15-69a2-4950-9bf5-c77868a86cfb-tmp\") pod \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " Oct 31 13:04:27.610345 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.610233 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-serving-cert\") pod \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " Oct 31 13:04:27.610345 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.610270 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-oauth-serving-cert\") pod \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " Oct 31 13:04:27.610345 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.610312 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-config\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-config\") pod \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " Oct 31 13:04:27.610345 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.610332 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-service-ca\") pod \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " Oct 31 13:04:27.610640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.610368 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d38ca15-69a2-4950-9bf5-c77868a86cfb-kube-api-access-ml8wx\" (UniqueName: \"kubernetes.io/projected/4d38ca15-69a2-4950-9bf5-c77868a86cfb-kube-api-access-ml8wx\") pod \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\" (UID: \"4d38ca15-69a2-4950-9bf5-c77868a86cfb\") " Oct 31 13:04:27.610640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.610484 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d38ca15-69a2-4950-9bf5-c77868a86cfb-tmp" pod "4d38ca15-69a2-4950-9bf5-c77868a86cfb" (UID: "4d38ca15-69a2-4950-9bf5-c77868a86cfb"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:27.610746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.610680 5532 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d38ca15-69a2-4950-9bf5-c77868a86cfb-tmp\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:27.610879 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.610857 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-oauth-serving-cert" pod "4d38ca15-69a2-4950-9bf5-c77868a86cfb" (UID: "4d38ca15-69a2-4950-9bf5-c77868a86cfb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:27.610936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.610851 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-service-ca" pod "4d38ca15-69a2-4950-9bf5-c77868a86cfb" (UID: "4d38ca15-69a2-4950-9bf5-c77868a86cfb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:27.610936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.610855 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-config" pod "4d38ca15-69a2-4950-9bf5-c77868a86cfb" (UID: "4d38ca15-69a2-4950-9bf5-c77868a86cfb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:27.612634 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.612613 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-serving-cert" pod "4d38ca15-69a2-4950-9bf5-c77868a86cfb" (UID: "4d38ca15-69a2-4950-9bf5-c77868a86cfb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:27.612724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.612641 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d38ca15-69a2-4950-9bf5-c77868a86cfb-kube-api-access-ml8wx" pod "4d38ca15-69a2-4950-9bf5-c77868a86cfb" (UID: "4d38ca15-69a2-4950-9bf5-c77868a86cfb"). InnerVolumeSpecName "kube-api-access-ml8wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:27.612772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.612748 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-oauth-config" pod "4d38ca15-69a2-4950-9bf5-c77868a86cfb" (UID: "4d38ca15-69a2-4950-9bf5-c77868a86cfb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:27.711048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.711013 5532 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-config\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:27.711048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.711041 5532 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:27.711224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.711052 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ml8wx\" (UniqueName: \"kubernetes.io/projected/4d38ca15-69a2-4950-9bf5-c77868a86cfb-kube-api-access-ml8wx\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:27.711224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.711064 5532 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-oauth-config\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:27.711224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.711073 5532 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d38ca15-69a2-4950-9bf5-c77868a86cfb-console-serving-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:27.711224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:27.711082 5532 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d38ca15-69a2-4950-9bf5-c77868a86cfb-oauth-serving-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:28.290763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:28.290684 5532 generic.go:356] "Generic (PLEG): container finished" podID="4d38ca15-69a2-4950-9bf5-c77868a86cfb" containerID="445f68b339fbb070e4bb7fa7195c1255bdc1bdc6ad63a526713ed16ff6c25231" exitCode=0 Oct 31 13:04:28.290763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:28.290740 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64885574d7-l7vzc" Oct 31 13:04:28.291188 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:28.290762 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64885574d7-l7vzc" event={"ID":"4d38ca15-69a2-4950-9bf5-c77868a86cfb","Type":"ContainerDied","Data":"445f68b339fbb070e4bb7fa7195c1255bdc1bdc6ad63a526713ed16ff6c25231"} Oct 31 13:04:28.291188 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:28.290824 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64885574d7-l7vzc" event={"ID":"4d38ca15-69a2-4950-9bf5-c77868a86cfb","Type":"ContainerDied","Data":"0db026a30d618fbaa7375a75aa984d06041c8e566648bb59d6e58e390eee588b"} Oct 31 13:04:28.291188 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:28.290896 5532 scope.go:122] "RemoveContainer" containerID="445f68b339fbb070e4bb7fa7195c1255bdc1bdc6ad63a526713ed16ff6c25231" Oct 31 13:04:28.303935 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:28.303919 5532 scope.go:122] "RemoveContainer" containerID="445f68b339fbb070e4bb7fa7195c1255bdc1bdc6ad63a526713ed16ff6c25231" Oct 31 13:04:28.304184 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:04:28.304165 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445f68b339fbb070e4bb7fa7195c1255bdc1bdc6ad63a526713ed16ff6c25231\": container with ID starting with 445f68b339fbb070e4bb7fa7195c1255bdc1bdc6ad63a526713ed16ff6c25231 not found: ID does not exist" containerID="445f68b339fbb070e4bb7fa7195c1255bdc1bdc6ad63a526713ed16ff6c25231" Oct 31 13:04:28.304254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:28.304196 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445f68b339fbb070e4bb7fa7195c1255bdc1bdc6ad63a526713ed16ff6c25231"} err="failed to get container status \"445f68b339fbb070e4bb7fa7195c1255bdc1bdc6ad63a526713ed16ff6c25231\": rpc error: code = NotFound desc = could not find container \"445f68b339fbb070e4bb7fa7195c1255bdc1bdc6ad63a526713ed16ff6c25231\": container with ID starting with 445f68b339fbb070e4bb7fa7195c1255bdc1bdc6ad63a526713ed16ff6c25231 not found: ID does not exist" Oct 31 13:04:28.312691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:28.312670 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-64885574d7-l7vzc"] Oct 31 13:04:28.317932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:28.317912 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-64885574d7-l7vzc"] Oct 31 13:04:30.000827 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:30.000792 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d38ca15-69a2-4950-9bf5-c77868a86cfb" path="/var/lib/kubelet/pods/4d38ca15-69a2-4950-9bf5-c77868a86cfb/volumes" Oct 31 13:04:31.760723 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:04:31.760692 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:04:32.206502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.206470 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-prddg-d22mn"] Oct 31 13:04:32.206730 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.206692 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-prddg-d22mn" podUID="ccf935cb-694a-4d54-b7a3-f0be4bc1212c" containerName="insights-gathering" containerID="cri-o://7972a59d25d88bce295d3bafbaa2543562203f9c595d50562c67dac3744b6338" gracePeriod=30 Oct 31 13:04:32.493659 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.493628 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:04:32.551316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.551283 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-trusted-ca-bundle\") pod \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " Oct 31 13:04:32.551316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.551321 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-kube-api-access-jfnsx\" (UniqueName: \"kubernetes.io/projected/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-kube-api-access-jfnsx\") pod \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " Oct 31 13:04:32.551501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.551340 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-service-ca-bundle\") pod \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " Oct 31 13:04:32.551501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.551434 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-archives-path\") pod \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\" (UID: \"ccf935cb-694a-4d54-b7a3-f0be4bc1212c\") " Oct 31 13:04:32.551765 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.551666 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-trusted-ca-bundle" pod "ccf935cb-694a-4d54-b7a3-f0be4bc1212c" (UID: "ccf935cb-694a-4d54-b7a3-f0be4bc1212c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:32.551835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.551785 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-archives-path" pod "ccf935cb-694a-4d54-b7a3-f0be4bc1212c" (UID: "ccf935cb-694a-4d54-b7a3-f0be4bc1212c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:32.551835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.551783 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-service-ca-bundle" pod "ccf935cb-694a-4d54-b7a3-f0be4bc1212c" (UID: "ccf935cb-694a-4d54-b7a3-f0be4bc1212c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:32.553552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.553532 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-kube-api-access-jfnsx" pod "ccf935cb-694a-4d54-b7a3-f0be4bc1212c" (UID: "ccf935cb-694a-4d54-b7a3-f0be4bc1212c"). InnerVolumeSpecName "kube-api-access-jfnsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:32.652539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.652504 5532 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-trusted-ca-bundle\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:32.652539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.652531 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfnsx\" (UniqueName: \"kubernetes.io/projected/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-kube-api-access-jfnsx\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:32.652539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.652542 5532 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-service-ca-bundle\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:32.652760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:32.652551 5532 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/ccf935cb-694a-4d54-b7a3-f0be4bc1212c-archives-path\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:33.313014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:33.312973 5532 generic.go:356] "Generic (PLEG): container finished" podID="ccf935cb-694a-4d54-b7a3-f0be4bc1212c" containerID="7972a59d25d88bce295d3bafbaa2543562203f9c595d50562c67dac3744b6338" exitCode=2 Oct 31 13:04:33.313448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:33.313029 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-prddg-d22mn" Oct 31 13:04:33.313448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:33.313059 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-prddg-d22mn" event={"ID":"ccf935cb-694a-4d54-b7a3-f0be4bc1212c","Type":"ContainerDied","Data":"7972a59d25d88bce295d3bafbaa2543562203f9c595d50562c67dac3744b6338"} Oct 31 13:04:33.313448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:33.313106 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-prddg-d22mn" event={"ID":"ccf935cb-694a-4d54-b7a3-f0be4bc1212c","Type":"ContainerDied","Data":"1dcd59a48b4c83fd45e6638c9df05b23c118bbfdcc28b472a48f59b8bf4e3f90"} Oct 31 13:04:33.313448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:33.313128 5532 scope.go:122] "RemoveContainer" containerID="7972a59d25d88bce295d3bafbaa2543562203f9c595d50562c67dac3744b6338" Oct 31 13:04:33.322462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:33.322445 5532 scope.go:122] "RemoveContainer" containerID="7972a59d25d88bce295d3bafbaa2543562203f9c595d50562c67dac3744b6338" Oct 31 13:04:33.322836 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:04:33.322813 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7972a59d25d88bce295d3bafbaa2543562203f9c595d50562c67dac3744b6338\": container with ID starting with 7972a59d25d88bce295d3bafbaa2543562203f9c595d50562c67dac3744b6338 not found: ID does not exist" containerID="7972a59d25d88bce295d3bafbaa2543562203f9c595d50562c67dac3744b6338" Oct 31 13:04:33.322937 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:33.322847 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7972a59d25d88bce295d3bafbaa2543562203f9c595d50562c67dac3744b6338"} err="failed to get container status \"7972a59d25d88bce295d3bafbaa2543562203f9c595d50562c67dac3744b6338\": rpc error: code = NotFound desc = could not find container \"7972a59d25d88bce295d3bafbaa2543562203f9c595d50562c67dac3744b6338\": container with ID starting with 7972a59d25d88bce295d3bafbaa2543562203f9c595d50562c67dac3744b6338 not found: ID does not exist" Oct 31 13:04:33.335685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:33.335657 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-prddg-d22mn"] Oct 31 13:04:33.339952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:33.339931 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-prddg-d22mn"] Oct 31 13:04:33.999942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:33.999910 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf935cb-694a-4d54-b7a3-f0be4bc1212c" path="/var/lib/kubelet/pods/ccf935cb-694a-4d54-b7a3-f0be4bc1212c/volumes" Oct 31 13:04:48.186136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:48.186092 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:04:48.231994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:48.231964 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:04:48.385227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:48.385199 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:04:57.878859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:57.878824 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 13:04:57.879541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:57.879104 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="alertmanager" containerID="cri-o://b374bf9aee74f40be27dd6864f78393762d9259c76f1cf7026c927f765535c44" gracePeriod=120 Oct 31 13:04:57.879541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:57.879130 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="kube-rbac-proxy-metric" containerID="cri-o://05da427806f0d54a530a3fbc4beca775c405034584b8604aa329939de1a59eb4" gracePeriod=120 Oct 31 13:04:57.879541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:57.879153 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="prom-label-proxy" containerID="cri-o://393beb865019774cebee9b24cb3e5da3673356faab44321dd6b846c68887f024" gracePeriod=120 Oct 31 13:04:57.879541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:57.879177 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="kube-rbac-proxy-web" containerID="cri-o://334adab89868ffd6c4c096259d08fac33f717760223dd42e143421cddfcff1ee" gracePeriod=120 Oct 31 13:04:57.879541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:57.879200 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="config-reloader" containerID="cri-o://fd46c35c975ae610e101333d0f86341a6f40c20f4986927bc914ff43f114d910" gracePeriod=120 Oct 31 13:04:57.879541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:57.879195 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="kube-rbac-proxy" containerID="cri-o://2100d067125a7c00cf540a5f2a668f44a2008b6afa0c8f8e2186f48c1c434a14" gracePeriod=120 Oct 31 13:04:58.407804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:58.407776 5532 generic.go:356] "Generic (PLEG): container finished" podID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerID="393beb865019774cebee9b24cb3e5da3673356faab44321dd6b846c68887f024" exitCode=0 Oct 31 13:04:58.407804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:58.407798 5532 generic.go:356] "Generic (PLEG): container finished" podID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerID="05da427806f0d54a530a3fbc4beca775c405034584b8604aa329939de1a59eb4" exitCode=0 Oct 31 13:04:58.407804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:58.407807 5532 generic.go:356] "Generic (PLEG): container finished" podID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerID="2100d067125a7c00cf540a5f2a668f44a2008b6afa0c8f8e2186f48c1c434a14" exitCode=0 Oct 31 13:04:58.407804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:58.407813 5532 generic.go:356] "Generic (PLEG): container finished" podID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerID="fd46c35c975ae610e101333d0f86341a6f40c20f4986927bc914ff43f114d910" exitCode=0 Oct 31 13:04:58.408048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:58.407819 5532 generic.go:356] "Generic (PLEG): container finished" podID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerID="b374bf9aee74f40be27dd6864f78393762d9259c76f1cf7026c927f765535c44" exitCode=0 Oct 31 13:04:58.408048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:58.407848 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerDied","Data":"393beb865019774cebee9b24cb3e5da3673356faab44321dd6b846c68887f024"} Oct 31 13:04:58.408048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:58.407912 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerDied","Data":"05da427806f0d54a530a3fbc4beca775c405034584b8604aa329939de1a59eb4"} Oct 31 13:04:58.408048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:58.407939 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerDied","Data":"2100d067125a7c00cf540a5f2a668f44a2008b6afa0c8f8e2186f48c1c434a14"} Oct 31 13:04:58.408048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:58.407962 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerDied","Data":"fd46c35c975ae610e101333d0f86341a6f40c20f4986927bc914ff43f114d910"} Oct 31 13:04:58.408048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:58.407984 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerDied","Data":"b374bf9aee74f40be27dd6864f78393762d9259c76f1cf7026c927f765535c44"} Oct 31 13:04:59.127913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.127883 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.192180 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192106 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-web\") pod \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " Oct 31 13:04:59.192180 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192162 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-tls-assets\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-tls-assets\") pod \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " Oct 31 13:04:59.192395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192240 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-web-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-web-config\") pod \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " Oct 31 13:04:59.192395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192280 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-metric\") pod \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " Oct 31 13:04:59.192395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192312 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-kube-api-access-xdn2q\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-kube-api-access-xdn2q\") pod \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " Oct 31 13:04:59.192395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192353 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-volume\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-volume\") pod \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " Oct 31 13:04:59.192395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192386 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-main-db\") pod \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " Oct 31 13:04:59.192674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192421 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-trusted-ca-bundle\") pod \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " Oct 31 13:04:59.192674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192452 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-metrics-client-ca\") pod \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " Oct 31 13:04:59.192674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192487 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-cluster-tls-config\") pod \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " Oct 31 13:04:59.192674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192528 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-out\") pod \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " Oct 31 13:04:59.192674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192563 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy\") pod \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " Oct 31 13:04:59.192674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192630 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-main-tls\") pod \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\" (UID: \"2b9c948c-9fd3-41e3-8815-208cfb56fe33\") " Oct 31 13:04:59.192972 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.192927 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-trusted-ca-bundle" pod "2b9c948c-9fd3-41e3-8815-208cfb56fe33" (UID: "2b9c948c-9fd3-41e3-8815-208cfb56fe33"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:59.193609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.193255 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-main-db" pod "2b9c948c-9fd3-41e3-8815-208cfb56fe33" (UID: "2b9c948c-9fd3-41e3-8815-208cfb56fe33"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:59.193609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.193537 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-metrics-client-ca" pod "2b9c948c-9fd3-41e3-8815-208cfb56fe33" (UID: "2b9c948c-9fd3-41e3-8815-208cfb56fe33"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:04:59.195517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.195477 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-metric" pod "2b9c948c-9fd3-41e3-8815-208cfb56fe33" (UID: "2b9c948c-9fd3-41e3-8815-208cfb56fe33"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:59.195870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.195832 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-main-tls" pod "2b9c948c-9fd3-41e3-8815-208cfb56fe33" (UID: "2b9c948c-9fd3-41e3-8815-208cfb56fe33"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:59.196084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.196024 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-tls-assets" pod "2b9c948c-9fd3-41e3-8815-208cfb56fe33" (UID: "2b9c948c-9fd3-41e3-8815-208cfb56fe33"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:59.196163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.196071 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-web" pod "2b9c948c-9fd3-41e3-8815-208cfb56fe33" (UID: "2b9c948c-9fd3-41e3-8815-208cfb56fe33"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:59.197184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.197154 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-volume" pod "2b9c948c-9fd3-41e3-8815-208cfb56fe33" (UID: "2b9c948c-9fd3-41e3-8815-208cfb56fe33"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:59.197616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.197577 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-kube-api-access-xdn2q" pod "2b9c948c-9fd3-41e3-8815-208cfb56fe33" (UID: "2b9c948c-9fd3-41e3-8815-208cfb56fe33"). InnerVolumeSpecName "kube-api-access-xdn2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:04:59.197740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.197716 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-out" pod "2b9c948c-9fd3-41e3-8815-208cfb56fe33" (UID: "2b9c948c-9fd3-41e3-8815-208cfb56fe33"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:04:59.198156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.198133 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy" pod "2b9c948c-9fd3-41e3-8815-208cfb56fe33" (UID: "2b9c948c-9fd3-41e3-8815-208cfb56fe33"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:59.201625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.201570 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-cluster-tls-config" pod "2b9c948c-9fd3-41e3-8815-208cfb56fe33" (UID: "2b9c948c-9fd3-41e3-8815-208cfb56fe33"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:59.208784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.208709 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-web-config" pod "2b9c948c-9fd3-41e3-8815-208cfb56fe33" (UID: "2b9c948c-9fd3-41e3-8815-208cfb56fe33"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:04:59.294068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.294035 5532 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-tls-assets\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:59.294068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.294065 5532 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-web-config\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:59.294253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.294077 5532 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:59.294253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.294090 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdn2q\" (UniqueName: \"kubernetes.io/projected/2b9c948c-9fd3-41e3-8815-208cfb56fe33-kube-api-access-xdn2q\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:59.294253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.294100 5532 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:59.294253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.294108 5532 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-main-db\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:59.294253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.294117 5532 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:59.294253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.294126 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2b9c948c-9fd3-41e3-8815-208cfb56fe33-metrics-client-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:59.294253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.294134 5532 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-cluster-tls-config\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:59.294253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.294142 5532 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2b9c948c-9fd3-41e3-8815-208cfb56fe33-config-out\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:59.294253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.294151 5532 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:59.294253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.294159 5532 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-main-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:59.294253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.294168 5532 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2b9c948c-9fd3-41e3-8815-208cfb56fe33-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:04:59.414112 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.414081 5532 generic.go:356] "Generic (PLEG): container finished" podID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerID="334adab89868ffd6c4c096259d08fac33f717760223dd42e143421cddfcff1ee" exitCode=0 Oct 31 13:04:59.414268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.414169 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.414268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.414175 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerDied","Data":"334adab89868ffd6c4c096259d08fac33f717760223dd42e143421cddfcff1ee"} Oct 31 13:04:59.414268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.414245 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2b9c948c-9fd3-41e3-8815-208cfb56fe33","Type":"ContainerDied","Data":"153a7b0a80ec64f63310fca9ec410e95d6c16ec1526a12f84ae675e1622405ac"} Oct 31 13:04:59.414367 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.414281 5532 scope.go:122] "RemoveContainer" containerID="393beb865019774cebee9b24cb3e5da3673356faab44321dd6b846c68887f024" Oct 31 13:04:59.422673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.422657 5532 scope.go:122] "RemoveContainer" containerID="05da427806f0d54a530a3fbc4beca775c405034584b8604aa329939de1a59eb4" Oct 31 13:04:59.429320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.429305 5532 scope.go:122] "RemoveContainer" containerID="2100d067125a7c00cf540a5f2a668f44a2008b6afa0c8f8e2186f48c1c434a14" Oct 31 13:04:59.435958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.435941 5532 scope.go:122] "RemoveContainer" containerID="334adab89868ffd6c4c096259d08fac33f717760223dd42e143421cddfcff1ee" Oct 31 13:04:59.439165 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.439145 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 13:04:59.444562 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.443570 5532 scope.go:122] "RemoveContainer" containerID="fd46c35c975ae610e101333d0f86341a6f40c20f4986927bc914ff43f114d910" Oct 31 13:04:59.444562 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.444041 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 13:04:59.450531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.450515 5532 scope.go:122] "RemoveContainer" containerID="b374bf9aee74f40be27dd6864f78393762d9259c76f1cf7026c927f765535c44" Oct 31 13:04:59.457263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.457246 5532 scope.go:122] "RemoveContainer" containerID="5597b8699084216d1ebb3f8a5221731bf5b762337e5d25896e643d857b9a8089" Oct 31 13:04:59.463863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.463847 5532 scope.go:122] "RemoveContainer" containerID="393beb865019774cebee9b24cb3e5da3673356faab44321dd6b846c68887f024" Oct 31 13:04:59.464112 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:04:59.464094 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"393beb865019774cebee9b24cb3e5da3673356faab44321dd6b846c68887f024\": container with ID starting with 393beb865019774cebee9b24cb3e5da3673356faab44321dd6b846c68887f024 not found: ID does not exist" containerID="393beb865019774cebee9b24cb3e5da3673356faab44321dd6b846c68887f024" Oct 31 13:04:59.464165 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.464119 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"393beb865019774cebee9b24cb3e5da3673356faab44321dd6b846c68887f024"} err="failed to get container status \"393beb865019774cebee9b24cb3e5da3673356faab44321dd6b846c68887f024\": rpc error: code = NotFound desc = could not find container \"393beb865019774cebee9b24cb3e5da3673356faab44321dd6b846c68887f024\": container with ID starting with 393beb865019774cebee9b24cb3e5da3673356faab44321dd6b846c68887f024 not found: ID does not exist" Oct 31 13:04:59.464165 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.464136 5532 scope.go:122] "RemoveContainer" containerID="05da427806f0d54a530a3fbc4beca775c405034584b8604aa329939de1a59eb4" Oct 31 13:04:59.464333 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:04:59.464318 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05da427806f0d54a530a3fbc4beca775c405034584b8604aa329939de1a59eb4\": container with ID starting with 05da427806f0d54a530a3fbc4beca775c405034584b8604aa329939de1a59eb4 not found: ID does not exist" containerID="05da427806f0d54a530a3fbc4beca775c405034584b8604aa329939de1a59eb4" Oct 31 13:04:59.464371 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.464337 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05da427806f0d54a530a3fbc4beca775c405034584b8604aa329939de1a59eb4"} err="failed to get container status \"05da427806f0d54a530a3fbc4beca775c405034584b8604aa329939de1a59eb4\": rpc error: code = NotFound desc = could not find container \"05da427806f0d54a530a3fbc4beca775c405034584b8604aa329939de1a59eb4\": container with ID starting with 05da427806f0d54a530a3fbc4beca775c405034584b8604aa329939de1a59eb4 not found: ID does not exist" Oct 31 13:04:59.464371 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.464353 5532 scope.go:122] "RemoveContainer" containerID="2100d067125a7c00cf540a5f2a668f44a2008b6afa0c8f8e2186f48c1c434a14" Oct 31 13:04:59.464569 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:04:59.464552 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2100d067125a7c00cf540a5f2a668f44a2008b6afa0c8f8e2186f48c1c434a14\": container with ID starting with 2100d067125a7c00cf540a5f2a668f44a2008b6afa0c8f8e2186f48c1c434a14 not found: ID does not exist" containerID="2100d067125a7c00cf540a5f2a668f44a2008b6afa0c8f8e2186f48c1c434a14" Oct 31 13:04:59.464634 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.464573 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2100d067125a7c00cf540a5f2a668f44a2008b6afa0c8f8e2186f48c1c434a14"} err="failed to get container status \"2100d067125a7c00cf540a5f2a668f44a2008b6afa0c8f8e2186f48c1c434a14\": rpc error: code = NotFound desc = could not find container \"2100d067125a7c00cf540a5f2a668f44a2008b6afa0c8f8e2186f48c1c434a14\": container with ID starting with 2100d067125a7c00cf540a5f2a668f44a2008b6afa0c8f8e2186f48c1c434a14 not found: ID does not exist" Oct 31 13:04:59.464634 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.464586 5532 scope.go:122] "RemoveContainer" containerID="334adab89868ffd6c4c096259d08fac33f717760223dd42e143421cddfcff1ee" Oct 31 13:04:59.464893 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:04:59.464878 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"334adab89868ffd6c4c096259d08fac33f717760223dd42e143421cddfcff1ee\": container with ID starting with 334adab89868ffd6c4c096259d08fac33f717760223dd42e143421cddfcff1ee not found: ID does not exist" containerID="334adab89868ffd6c4c096259d08fac33f717760223dd42e143421cddfcff1ee" Oct 31 13:04:59.464932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.464896 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"334adab89868ffd6c4c096259d08fac33f717760223dd42e143421cddfcff1ee"} err="failed to get container status \"334adab89868ffd6c4c096259d08fac33f717760223dd42e143421cddfcff1ee\": rpc error: code = NotFound desc = could not find container \"334adab89868ffd6c4c096259d08fac33f717760223dd42e143421cddfcff1ee\": container with ID starting with 334adab89868ffd6c4c096259d08fac33f717760223dd42e143421cddfcff1ee not found: ID does not exist" Oct 31 13:04:59.464932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.464909 5532 scope.go:122] "RemoveContainer" containerID="fd46c35c975ae610e101333d0f86341a6f40c20f4986927bc914ff43f114d910" Oct 31 13:04:59.465114 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:04:59.465101 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd46c35c975ae610e101333d0f86341a6f40c20f4986927bc914ff43f114d910\": container with ID starting with fd46c35c975ae610e101333d0f86341a6f40c20f4986927bc914ff43f114d910 not found: ID does not exist" containerID="fd46c35c975ae610e101333d0f86341a6f40c20f4986927bc914ff43f114d910" Oct 31 13:04:59.465153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.465118 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd46c35c975ae610e101333d0f86341a6f40c20f4986927bc914ff43f114d910"} err="failed to get container status \"fd46c35c975ae610e101333d0f86341a6f40c20f4986927bc914ff43f114d910\": rpc error: code = NotFound desc = could not find container \"fd46c35c975ae610e101333d0f86341a6f40c20f4986927bc914ff43f114d910\": container with ID starting with fd46c35c975ae610e101333d0f86341a6f40c20f4986927bc914ff43f114d910 not found: ID does not exist" Oct 31 13:04:59.465153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.465130 5532 scope.go:122] "RemoveContainer" containerID="b374bf9aee74f40be27dd6864f78393762d9259c76f1cf7026c927f765535c44" Oct 31 13:04:59.465310 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:04:59.465295 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b374bf9aee74f40be27dd6864f78393762d9259c76f1cf7026c927f765535c44\": container with ID starting with b374bf9aee74f40be27dd6864f78393762d9259c76f1cf7026c927f765535c44 not found: ID does not exist" containerID="b374bf9aee74f40be27dd6864f78393762d9259c76f1cf7026c927f765535c44" Oct 31 13:04:59.465346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.465313 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b374bf9aee74f40be27dd6864f78393762d9259c76f1cf7026c927f765535c44"} err="failed to get container status \"b374bf9aee74f40be27dd6864f78393762d9259c76f1cf7026c927f765535c44\": rpc error: code = NotFound desc = could not find container \"b374bf9aee74f40be27dd6864f78393762d9259c76f1cf7026c927f765535c44\": container with ID starting with b374bf9aee74f40be27dd6864f78393762d9259c76f1cf7026c927f765535c44 not found: ID does not exist" Oct 31 13:04:59.465346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.465324 5532 scope.go:122] "RemoveContainer" containerID="5597b8699084216d1ebb3f8a5221731bf5b762337e5d25896e643d857b9a8089" Oct 31 13:04:59.465499 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:04:59.465486 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5597b8699084216d1ebb3f8a5221731bf5b762337e5d25896e643d857b9a8089\": container with ID starting with 5597b8699084216d1ebb3f8a5221731bf5b762337e5d25896e643d857b9a8089 not found: ID does not exist" containerID="5597b8699084216d1ebb3f8a5221731bf5b762337e5d25896e643d857b9a8089" Oct 31 13:04:59.465542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.465501 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5597b8699084216d1ebb3f8a5221731bf5b762337e5d25896e643d857b9a8089"} err="failed to get container status \"5597b8699084216d1ebb3f8a5221731bf5b762337e5d25896e643d857b9a8089\": rpc error: code = NotFound desc = could not find container \"5597b8699084216d1ebb3f8a5221731bf5b762337e5d25896e643d857b9a8089\": container with ID starting with 5597b8699084216d1ebb3f8a5221731bf5b762337e5d25896e643d857b9a8089 not found: ID does not exist" Oct 31 13:04:59.479574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.479550 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 13:04:59.479973 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.479951 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ccf935cb-694a-4d54-b7a3-f0be4bc1212c" containerName="insights-gathering" Oct 31 13:04:59.480077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480000 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf935cb-694a-4d54-b7a3-f0be4bc1212c" containerName="insights-gathering" Oct 31 13:04:59.480077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480035 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4d38ca15-69a2-4950-9bf5-c77868a86cfb" containerName="console" Oct 31 13:04:59.480077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480043 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d38ca15-69a2-4950-9bf5-c77868a86cfb" containerName="console" Oct 31 13:04:59.480077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480055 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="kube-rbac-proxy" Oct 31 13:04:59.480077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480062 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="kube-rbac-proxy" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480081 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="alertmanager" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480088 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="alertmanager" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480101 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="prom-label-proxy" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480108 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="prom-label-proxy" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480127 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="config-reloader" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480134 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="config-reloader" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480143 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="init-config-reloader" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480150 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="init-config-reloader" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480172 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="kube-rbac-proxy-web" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480179 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="kube-rbac-proxy-web" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480187 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="kube-rbac-proxy-metric" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480194 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="kube-rbac-proxy-metric" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480298 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d38ca15-69a2-4950-9bf5-c77868a86cfb" containerName="console" Oct 31 13:04:59.480311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480309 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="kube-rbac-proxy" Oct 31 13:04:59.480993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480320 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="alertmanager" Oct 31 13:04:59.480993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480328 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="prom-label-proxy" Oct 31 13:04:59.480993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480338 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="config-reloader" Oct 31 13:04:59.480993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480349 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="kube-rbac-proxy-web" Oct 31 13:04:59.480993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480360 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" containerName="kube-rbac-proxy-metric" Oct 31 13:04:59.480993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.480371 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf935cb-694a-4d54-b7a3-f0be4bc1212c" containerName="insights-gathering" Oct 31 13:04:59.483619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.483586 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.488489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.488460 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 31 13:04:59.488697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.488527 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 31 13:04:59.488697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.488541 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 31 13:04:59.488697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.488551 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 31 13:04:59.488697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.488583 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 31 13:04:59.488697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.488610 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 31 13:04:59.488697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.488622 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 31 13:04:59.488697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.488612 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 31 13:04:59.488697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.488675 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-nztnk\"" Oct 31 13:04:59.492289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.492272 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 31 13:04:59.503696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.503678 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 13:04:59.596965 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.596927 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8wwr\" (UniqueName: \"kubernetes.io/projected/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-kube-api-access-p8wwr\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.596965 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.596968 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.597195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.596985 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.597195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.597143 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.597195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.597179 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.597339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.597197 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-web-config\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.597339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.597236 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.597443 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.597336 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.597443 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.597380 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.597507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.597456 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.597507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.597474 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.597507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.597494 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-config-out\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.597649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.597510 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.698822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.698761 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.698822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.698800 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.698822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.698820 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.699024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.698939 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-config-out\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.699024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.698975 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.699125 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.699023 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8wwr\" (UniqueName: \"kubernetes.io/projected/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-kube-api-access-p8wwr\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.699125 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.699057 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.699125 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.699084 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.699278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.699141 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.699278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.699164 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.699278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.699186 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-web-config\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.699278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.699211 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.699278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.699250 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.699649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.699625 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.699810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.699785 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.701842 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.701813 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-config-out\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.702043 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.702017 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-tls-assets\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.702201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.702174 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.702324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.702167 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.702324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.702215 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.702324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.702261 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.702324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.702287 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.702519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.702431 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.702816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.702793 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-web-config\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.703913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.703896 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-config-volume\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.712186 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.712168 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8wwr\" (UniqueName: \"kubernetes.io/projected/bd1ae85c-6bcf-43cc-9709-3102e4c5bc59-kube-api-access-p8wwr\") pod \"alertmanager-main-0\" (UID: \"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59\") " pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:04:59.800215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:04:59.800183 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 31 13:05:00.000209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:00.000177 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9c948c-9fd3-41e3-8815-208cfb56fe33" path="/var/lib/kubelet/pods/2b9c948c-9fd3-41e3-8815-208cfb56fe33/volumes" Oct 31 13:05:00.084259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:00.084185 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 31 13:05:00.087791 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:05:00.087755 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd1ae85c_6bcf_43cc_9709_3102e4c5bc59.slice/crio-543739f54a9d1e1a6166ee9f4a68a1243eb476fa8e20d9ce8d432491dad15957 WatchSource:0}: Error finding container 543739f54a9d1e1a6166ee9f4a68a1243eb476fa8e20d9ce8d432491dad15957: Status 404 returned error can't find the container with id 543739f54a9d1e1a6166ee9f4a68a1243eb476fa8e20d9ce8d432491dad15957 Oct 31 13:05:00.418886 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:00.418852 5532 generic.go:356] "Generic (PLEG): container finished" podID="bd1ae85c-6bcf-43cc-9709-3102e4c5bc59" containerID="1e4ed71a6f8f4e3ecfa4ca8f0ad8f604b865cf15b1a244a631f7cc5a717252bd" exitCode=0 Oct 31 13:05:00.419294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:00.418933 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59","Type":"ContainerDied","Data":"1e4ed71a6f8f4e3ecfa4ca8f0ad8f604b865cf15b1a244a631f7cc5a717252bd"} Oct 31 13:05:00.419294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:00.419032 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59","Type":"ContainerStarted","Data":"543739f54a9d1e1a6166ee9f4a68a1243eb476fa8e20d9ce8d432491dad15957"} Oct 31 13:05:01.429524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.429486 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59","Type":"ContainerStarted","Data":"4bc65928c8daf2ce59fc5b023c2769c6082dbd60a1e1aa3d1830c444716313fe"} Oct 31 13:05:01.430032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.429568 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59","Type":"ContainerStarted","Data":"28ef7b094bda7f3c3c6ad4a45387eb1ea603d62dfd9f65bf2e9fa5e67b613544"} Oct 31 13:05:01.430032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.429630 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59","Type":"ContainerStarted","Data":"7ce9507c82814858e581a60b6515daf149a976a6b245309c7bc8201ff6ec0a9d"} Oct 31 13:05:01.430032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.429672 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59","Type":"ContainerStarted","Data":"9ff6a12b49ab2b2b00a89b2ca4f33f6742380b52b50455096171d2cb8261ff1e"} Oct 31 13:05:01.430032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.429714 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59","Type":"ContainerStarted","Data":"e5c8532b586755e4107df9d937443ffa1b9cf5a50f8e3f27f3fcf74de91d5f4d"} Oct 31 13:05:01.430032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.429754 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"bd1ae85c-6bcf-43cc-9709-3102e4c5bc59","Type":"ContainerStarted","Data":"13492c76755765412ad03b909951882b45c4cbf12536e9a932095e2ad6203027"} Oct 31 13:05:01.467036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.466984 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.455504612 podStartE2EDuration="2.466968441s" totalImagesPullingTime="0s" totalInitContainerRuntime="11.463829ms" isStatefulPod=false podCreationTimestamp="2026-10-31 13:04:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:01.46516514 +0000 UTC m=+69.610909007" watchObservedRunningTime="2026-10-31 13:05:01.466968441 +0000 UTC m=+69.612712292" Oct 31 13:05:01.760310 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:05:01.760231 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:05:01.893453 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.893412 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-96946754f-9jtzl"] Oct 31 13:05:01.896102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.896081 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:01.902049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.902029 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 31 13:05:01.905104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.905083 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 31 13:05:01.913575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.913553 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-96946754f-9jtzl"] Oct 31 13:05:01.985335 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.985306 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 31 13:05:01.985498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.985356 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 31 13:05:01.985498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.985393 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 31 13:05:01.985639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.985508 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-4zwnm\"" Oct 31 13:05:01.985639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:01.985518 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 31 13:05:02.023758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.023693 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddd6aed9-e9c4-4403-a941-bcb9020faac0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.023758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.023724 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddd6aed9-e9c4-4403-a941-bcb9020faac0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.023758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.023746 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddd6aed9-e9c4-4403-a941-bcb9020faac0-secret-telemeter-client\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.023940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.023818 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddd6aed9-e9c4-4403-a941-bcb9020faac0-telemeter-client-tls\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.023940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.023852 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddd6aed9-e9c4-4403-a941-bcb9020faac0-federate-client-tls\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.024009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.023955 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddd6aed9-e9c4-4403-a941-bcb9020faac0-metrics-client-ca\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.024009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.024000 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddd6aed9-e9c4-4403-a941-bcb9020faac0-serving-certs-ca-bundle\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.024070 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.024058 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcg82\" (UniqueName: \"kubernetes.io/projected/ddd6aed9-e9c4-4403-a941-bcb9020faac0-kube-api-access-rcg82\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.124872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.124836 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddd6aed9-e9c4-4403-a941-bcb9020faac0-serving-certs-ca-bundle\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.124872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.124877 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcg82\" (UniqueName: \"kubernetes.io/projected/ddd6aed9-e9c4-4403-a941-bcb9020faac0-kube-api-access-rcg82\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.125134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.124905 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddd6aed9-e9c4-4403-a941-bcb9020faac0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.125134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.124926 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddd6aed9-e9c4-4403-a941-bcb9020faac0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.125134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.124946 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddd6aed9-e9c4-4403-a941-bcb9020faac0-secret-telemeter-client\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.125134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.124968 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddd6aed9-e9c4-4403-a941-bcb9020faac0-telemeter-client-tls\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.125134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.125003 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddd6aed9-e9c4-4403-a941-bcb9020faac0-federate-client-tls\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.125134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.125058 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddd6aed9-e9c4-4403-a941-bcb9020faac0-metrics-client-ca\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.125831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.125801 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ddd6aed9-e9c4-4403-a941-bcb9020faac0-metrics-client-ca\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.125831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.125815 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddd6aed9-e9c4-4403-a941-bcb9020faac0-serving-certs-ca-bundle\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.126008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.125985 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ddd6aed9-e9c4-4403-a941-bcb9020faac0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.127679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.127651 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ddd6aed9-e9c4-4403-a941-bcb9020faac0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.127775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.127690 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/ddd6aed9-e9c4-4403-a941-bcb9020faac0-secret-telemeter-client\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.127839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.127785 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/ddd6aed9-e9c4-4403-a941-bcb9020faac0-telemeter-client-tls\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.127881 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.127852 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/ddd6aed9-e9c4-4403-a941-bcb9020faac0-federate-client-tls\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.138569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.138546 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcg82\" (UniqueName: \"kubernetes.io/projected/ddd6aed9-e9c4-4403-a941-bcb9020faac0-kube-api-access-rcg82\") pod \"telemeter-client-96946754f-9jtzl\" (UID: \"ddd6aed9-e9c4-4403-a941-bcb9020faac0\") " pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.209412 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.209381 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" Oct 31 13:05:02.304735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.304647 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:05:02.305134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.305099 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="prometheus" containerID="cri-o://1eb0a60c3ac6bafee09671af8bd29e30ba5e3fbed611e5a74ca08e637c02a3b5" gracePeriod=600 Oct 31 13:05:02.305271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.305195 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="kube-rbac-proxy" containerID="cri-o://d10cae6eda6aa669c0188cb8e44f3f26c310801ef231d630a93a70b85266459e" gracePeriod=600 Oct 31 13:05:02.305341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.305274 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="config-reloader" containerID="cri-o://85ebe3b76ef5a1a9cc3d0bff52702bbc61abf879559cef44ad0698322963ad3d" gracePeriod=600 Oct 31 13:05:02.305341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.305261 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="thanos-sidecar" containerID="cri-o://d321d37eef3a14597902e72643ba03d971884338fc517669c654ec9f3af79861" gracePeriod=600 Oct 31 13:05:02.305455 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.305365 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="kube-rbac-proxy-thanos" containerID="cri-o://487b4a83f2f5f401fbcf9521d948de749d34abe579934698633a22edd420c616" gracePeriod=600 Oct 31 13:05:02.305455 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.305376 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="kube-rbac-proxy-web" containerID="cri-o://5efcf5d9875c1ee6550c100c7b743286e4f846023aefde0ad98f5a608c41c9d8" gracePeriod=600 Oct 31 13:05:02.436647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.436620 5532 generic.go:356] "Generic (PLEG): container finished" podID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerID="487b4a83f2f5f401fbcf9521d948de749d34abe579934698633a22edd420c616" exitCode=0 Oct 31 13:05:02.436647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.436646 5532 generic.go:356] "Generic (PLEG): container finished" podID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerID="d10cae6eda6aa669c0188cb8e44f3f26c310801ef231d630a93a70b85266459e" exitCode=0 Oct 31 13:05:02.437086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.436657 5532 generic.go:356] "Generic (PLEG): container finished" podID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerID="d321d37eef3a14597902e72643ba03d971884338fc517669c654ec9f3af79861" exitCode=0 Oct 31 13:05:02.437086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.436665 5532 generic.go:356] "Generic (PLEG): container finished" podID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerID="85ebe3b76ef5a1a9cc3d0bff52702bbc61abf879559cef44ad0698322963ad3d" exitCode=0 Oct 31 13:05:02.437086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.436673 5532 generic.go:356] "Generic (PLEG): container finished" podID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerID="1eb0a60c3ac6bafee09671af8bd29e30ba5e3fbed611e5a74ca08e637c02a3b5" exitCode=0 Oct 31 13:05:02.437086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.436679 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerDied","Data":"487b4a83f2f5f401fbcf9521d948de749d34abe579934698633a22edd420c616"} Oct 31 13:05:02.437086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.436763 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerDied","Data":"d10cae6eda6aa669c0188cb8e44f3f26c310801ef231d630a93a70b85266459e"} Oct 31 13:05:02.437086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.436789 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerDied","Data":"d321d37eef3a14597902e72643ba03d971884338fc517669c654ec9f3af79861"} Oct 31 13:05:02.437086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.436814 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerDied","Data":"85ebe3b76ef5a1a9cc3d0bff52702bbc61abf879559cef44ad0698322963ad3d"} Oct 31 13:05:02.437086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.436839 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerDied","Data":"1eb0a60c3ac6bafee09671af8bd29e30ba5e3fbed611e5a74ca08e637c02a3b5"} Oct 31 13:05:02.497217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:02.497093 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-96946754f-9jtzl"] Oct 31 13:05:02.499251 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:05:02.499227 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd6aed9_e9c4_4403_a941_bcb9020faac0.slice/crio-25f5919a1ccf9e14e0201f6b07684e955eb4f8484709061f34f9aa93e706163e WatchSource:0}: Error finding container 25f5919a1ccf9e14e0201f6b07684e955eb4f8484709061f34f9aa93e706163e: Status 404 returned error can't find the container with id 25f5919a1ccf9e14e0201f6b07684e955eb4f8484709061f34f9aa93e706163e Oct 31 13:05:03.369909 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:05:03.369868 5532 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="1eb0a60c3ac6bafee09671af8bd29e30ba5e3fbed611e5a74ca08e637c02a3b5" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 13:05:03.370068 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:05:03.369930 5532 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="1eb0a60c3ac6bafee09671af8bd29e30ba5e3fbed611e5a74ca08e637c02a3b5" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 13:05:03.370068 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:05:03.369946 5532 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="1eb0a60c3ac6bafee09671af8bd29e30ba5e3fbed611e5a74ca08e637c02a3b5" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"] Oct 31 13:05:03.370068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.369965 5532 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled" Oct 31 13:05:03.443524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.442982 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" event={"ID":"ddd6aed9-e9c4-4403-a941-bcb9020faac0","Type":"ContainerStarted","Data":"25f5919a1ccf9e14e0201f6b07684e955eb4f8484709061f34f9aa93e706163e"} Oct 31 13:05:03.447195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.447165 5532 generic.go:356] "Generic (PLEG): container finished" podID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerID="5efcf5d9875c1ee6550c100c7b743286e4f846023aefde0ad98f5a608c41c9d8" exitCode=0 Oct 31 13:05:03.447308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.447227 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerDied","Data":"5efcf5d9875c1ee6550c100c7b743286e4f846023aefde0ad98f5a608c41c9d8"} Oct 31 13:05:03.554077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.554041 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:03.637206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637174 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-2\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637226 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637336 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-0\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637381 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637415 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-metrics-client-ca\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637447 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-serving-certs-ca-bundle\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637494 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-db\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637528 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-kubelet-serving-ca-bundle\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637552 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-2" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:03.637853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637555 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-tls\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637644 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-kube-rbac-proxy\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637680 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-config-out\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637704 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-1\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637723 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-grpc-tls\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637774 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-web-config\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-web-config\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637807 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-kube-api-access-5dcwc\" (UniqueName: \"kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-kube-api-access-5dcwc\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.637853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637849 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-metrics-client-certs\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.638277 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637883 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-tls-assets\" (UniqueName: \"kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-tls-assets\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.638277 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637931 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-trusted-ca-bundle\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.638277 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637968 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-config\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-config\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.638277 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.637999 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-serving-certs-ca-bundle" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:03.638277 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.638009 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-metrics-client-ca" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:03.638277 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.638007 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-thanos-prometheus-http-client-file\") pod \"c50430d1-c382-4a45-9672-d4cb509f8aaf\" (UID: \"c50430d1-c382-4a45-9672-d4cb509f8aaf\") " Oct 31 13:05:03.639936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.638330 5532 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-metrics-client-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.639936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.638353 5532 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.639936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.638370 5532 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.639936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.638771 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-kubelet-serving-ca-bundle" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:03.639936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.639409 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-1" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:03.641319 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.640856 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-db" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:03.641319 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.641042 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-trusted-ca-bundle" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:03.641319 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.641105 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-thanos-sidecar-tls" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:03.641319 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.641247 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-kube-rbac-proxy-web" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:03.641735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.641707 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-0" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:03.641813 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.641789 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-kube-rbac-proxy" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:03.642206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.642176 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-thanos-prometheus-http-client-file" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:03.643255 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.643218 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-tls-assets" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:03.643612 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.643559 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-tls" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:03.643827 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.643801 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-config" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:03.643909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.643875 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-config-out" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:03.644198 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.644174 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-metrics-client-certs" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:03.644265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.644193 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-kube-api-access-5dcwc" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "kube-api-access-5dcwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:03.645253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.645226 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-grpc-tls" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:03.658847 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.658819 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-web-config" pod "c50430d1-c382-4a45-9672-d4cb509f8aaf" (UID: "c50430d1-c382-4a45-9672-d4cb509f8aaf"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:03.739221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739137 5532 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-db\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739171 5532 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739186 5532 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739199 5532 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-kube-rbac-proxy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739211 5532 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c50430d1-c382-4a45-9672-d4cb509f8aaf-config-out\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739224 5532 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739238 5532 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-grpc-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739250 5532 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-web-config\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739262 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dcwc\" (UniqueName: \"kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-kube-api-access-5dcwc\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739274 5532 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-metrics-client-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739285 5532 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c50430d1-c382-4a45-9672-d4cb509f8aaf-tls-assets\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739298 5532 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-trusted-ca-bundle\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739312 5532 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-config\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739324 5532 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-thanos-prometheus-http-client-file\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739337 5532 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739350 5532 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c50430d1-c382-4a45-9672-d4cb509f8aaf-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:03.739678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:03.739363 5532 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c50430d1-c382-4a45-9672-d4cb509f8aaf-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:04.454728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.454692 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"c50430d1-c382-4a45-9672-d4cb509f8aaf","Type":"ContainerDied","Data":"11f17d2cc472af966598a715bd1c5677343800f4fab9d87543014b51364bf583"} Oct 31 13:05:04.454728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.454728 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.455272 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.454780 5532 scope.go:122] "RemoveContainer" containerID="487b4a83f2f5f401fbcf9521d948de749d34abe579934698633a22edd420c616" Oct 31 13:05:04.462331 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.462308 5532 scope.go:122] "RemoveContainer" containerID="d10cae6eda6aa669c0188cb8e44f3f26c310801ef231d630a93a70b85266459e" Oct 31 13:05:04.469410 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.469391 5532 scope.go:122] "RemoveContainer" containerID="5efcf5d9875c1ee6550c100c7b743286e4f846023aefde0ad98f5a608c41c9d8" Oct 31 13:05:04.479775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.479372 5532 scope.go:122] "RemoveContainer" containerID="d321d37eef3a14597902e72643ba03d971884338fc517669c654ec9f3af79861" Oct 31 13:05:04.481789 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.481764 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:05:04.486920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.486871 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:05:04.490313 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.490292 5532 scope.go:122] "RemoveContainer" containerID="85ebe3b76ef5a1a9cc3d0bff52702bbc61abf879559cef44ad0698322963ad3d" Oct 31 13:05:04.497054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.497038 5532 scope.go:122] "RemoveContainer" containerID="1eb0a60c3ac6bafee09671af8bd29e30ba5e3fbed611e5a74ca08e637c02a3b5" Oct 31 13:05:04.504034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.504020 5532 scope.go:122] "RemoveContainer" containerID="83ed47794c080bd2a987bdfc04eda6dfc1f6304c1af1a46b237ea2e03031c068" Oct 31 13:05:04.525033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525012 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:05:04.525322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525310 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="kube-rbac-proxy" Oct 31 13:05:04.525362 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525324 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="kube-rbac-proxy" Oct 31 13:05:04.525362 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525332 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="kube-rbac-proxy-thanos" Oct 31 13:05:04.525362 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525337 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="kube-rbac-proxy-thanos" Oct 31 13:05:04.525362 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525355 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="config-reloader" Oct 31 13:05:04.525507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525363 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="config-reloader" Oct 31 13:05:04.525507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525394 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="thanos-sidecar" Oct 31 13:05:04.525507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525399 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="thanos-sidecar" Oct 31 13:05:04.525507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525410 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="prometheus" Oct 31 13:05:04.525507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525415 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="prometheus" Oct 31 13:05:04.525507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525421 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="kube-rbac-proxy-web" Oct 31 13:05:04.525507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525428 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="kube-rbac-proxy-web" Oct 31 13:05:04.525507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525439 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="init-config-reloader" Oct 31 13:05:04.525507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525444 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="init-config-reloader" Oct 31 13:05:04.525507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525498 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="prometheus" Oct 31 13:05:04.525507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525508 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="kube-rbac-proxy-web" Oct 31 13:05:04.525857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525522 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="kube-rbac-proxy" Oct 31 13:05:04.525857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525528 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="kube-rbac-proxy-thanos" Oct 31 13:05:04.525857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525535 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="config-reloader" Oct 31 13:05:04.525857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.525541 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" containerName="thanos-sidecar" Oct 31 13:05:04.535073 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.535053 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.541341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.541294 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 31 13:05:04.541341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.541296 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 31 13:05:04.541491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.541371 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 31 13:05:04.541491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.541407 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 31 13:05:04.541782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.541758 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 31 13:05:04.541871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.541765 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 31 13:05:04.541871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.541785 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 31 13:05:04.541871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.541796 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-a7uov9imvc59d\"" Oct 31 13:05:04.541871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.541788 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 31 13:05:04.541871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.541813 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 31 13:05:04.542040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.541955 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 31 13:05:04.542040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.541966 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 31 13:05:04.542040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.542017 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 31 13:05:04.542135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.542073 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-896z6\"" Oct 31 13:05:04.543370 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.543339 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 31 13:05:04.546751 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.546712 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 31 13:05:04.546841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.546799 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 31 13:05:04.552431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.552409 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:05:04.646394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646315 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.646394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646368 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-config\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.646658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646462 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.646658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646501 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.646658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646535 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.646658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646610 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.646658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646653 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6xh2\" (UniqueName: \"kubernetes.io/projected/0e683f34-6fee-4e68-a479-6688ab8221c3-kube-api-access-l6xh2\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.646855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646711 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e683f34-6fee-4e68-a479-6688ab8221c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.646855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646743 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.646855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646802 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.646855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646823 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.646855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646841 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.647041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646863 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.647041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646938 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.647041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646981 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.647041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.646997 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e683f34-6fee-4e68-a479-6688ab8221c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.647041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.647018 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.647041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.647039 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.647338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.647107 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.647338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.647151 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.748696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.748615 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.748696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.748655 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.748696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.748674 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.748954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.748704 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.748954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.748729 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e683f34-6fee-4e68-a479-6688ab8221c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.748954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.748752 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.748954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.748774 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.748954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.748817 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.748954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.748843 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.748954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.748888 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.748954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.748909 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-config\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.748954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.748945 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.749306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.748987 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.749306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.749013 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.749306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.749037 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.749306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.749058 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6xh2\" (UniqueName: \"kubernetes.io/projected/0e683f34-6fee-4e68-a479-6688ab8221c3-kube-api-access-l6xh2\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.749306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.749081 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e683f34-6fee-4e68-a479-6688ab8221c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.749306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.749105 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.749306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.749131 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.749306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.749156 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.749711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.749616 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.749767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.749705 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.749767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.749732 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.749879 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.749848 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.750171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.750142 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.751145 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.751118 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.751896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.751868 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.751978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.751917 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.752905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.752574 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0e683f34-6fee-4e68-a479-6688ab8221c3-config-out\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.753056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.753031 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.753371 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.753343 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-web-config\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.753892 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.753831 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.754012 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.753986 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0e683f34-6fee-4e68-a479-6688ab8221c3-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.754339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.754318 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.755295 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.755273 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.755391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.755310 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/0e683f34-6fee-4e68-a479-6688ab8221c3-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.755535 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.755517 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.755964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.755941 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-config\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.756473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.756457 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0e683f34-6fee-4e68-a479-6688ab8221c3-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.763790 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.763773 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6xh2\" (UniqueName: \"kubernetes.io/projected/0e683f34-6fee-4e68-a479-6688ab8221c3-kube-api-access-l6xh2\") pod \"prometheus-k8s-0\" (UID: \"0e683f34-6fee-4e68-a479-6688ab8221c3\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:04.856446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:04.856396 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:05.153415 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.153379 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 31 13:05:05.157979 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:05:05.157954 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e683f34_6fee_4e68_a479_6688ab8221c3.slice/crio-9c3ae8b309fbca2fc36a596329cf1241308933d1585471223e29b699be81e9e0 WatchSource:0}: Error finding container 9c3ae8b309fbca2fc36a596329cf1241308933d1585471223e29b699be81e9e0: Status 404 returned error can't find the container with id 9c3ae8b309fbca2fc36a596329cf1241308933d1585471223e29b699be81e9e0 Oct 31 13:05:05.460574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.460540 5532 generic.go:356] "Generic (PLEG): container finished" podID="0e683f34-6fee-4e68-a479-6688ab8221c3" containerID="170bf39e43b62ae458b929916d92aca2d66dde7fc312f0e8f555baa7c855edc7" exitCode=0 Oct 31 13:05:05.461069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.460629 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e683f34-6fee-4e68-a479-6688ab8221c3","Type":"ContainerDied","Data":"170bf39e43b62ae458b929916d92aca2d66dde7fc312f0e8f555baa7c855edc7"} Oct 31 13:05:05.461069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.460714 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e683f34-6fee-4e68-a479-6688ab8221c3","Type":"ContainerStarted","Data":"9c3ae8b309fbca2fc36a596329cf1241308933d1585471223e29b699be81e9e0"} Oct 31 13:05:05.463004 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.462977 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" event={"ID":"ddd6aed9-e9c4-4403-a941-bcb9020faac0","Type":"ContainerStarted","Data":"06dd7871fd2d17defbd84cf30592871d1e5064314bda28b9358b908eba6a6846"} Oct 31 13:05:05.463104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.463031 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" event={"ID":"ddd6aed9-e9c4-4403-a941-bcb9020faac0","Type":"ContainerStarted","Data":"2b37ad0f5f5ff61b2c5c1320741a053a879773ef2ef54f371d9c0ca309e4c822"} Oct 31 13:05:05.463104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.463066 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" event={"ID":"ddd6aed9-e9c4-4403-a941-bcb9020faac0","Type":"ContainerStarted","Data":"57fc88d041e40f76570be7bec5596fa53d092cad58a060172c31eef0a4e14d0c"} Oct 31 13:05:05.523197 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.523151 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-96946754f-9jtzl" podStartSLOduration=1.99144927 podStartE2EDuration="4.523138212s" totalImagesPullingTime="2.531688942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:05.522550353 +0000 UTC m=+73.668294185" watchObservedRunningTime="2026-10-31 13:05:05.523138212 +0000 UTC m=+73.668882085" Oct 31 13:05:05.624633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.624573 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5f48f6db46-tjpqf"] Oct 31 13:05:05.628691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.628670 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.640853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.640820 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f48f6db46-tjpqf"] Oct 31 13:05:05.757241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.756826 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-config\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.757241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.756893 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-serving-cert\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.757241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.756929 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ttq2\" (UniqueName: \"kubernetes.io/projected/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-kube-api-access-6ttq2\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.757241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.756957 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-trusted-ca-bundle\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.757241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.756983 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-oauth-config\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.757241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.757019 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-tmp\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.757241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.757050 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-oauth-serving-cert\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.757241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.757090 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-service-ca\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.857942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.857916 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-serving-cert\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.858061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.857954 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ttq2\" (UniqueName: \"kubernetes.io/projected/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-kube-api-access-6ttq2\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.858061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.857974 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-trusted-ca-bundle\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.858061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.857990 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-oauth-config\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.858061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.858012 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-tmp\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.858061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.858041 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-oauth-serving-cert\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.858226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.858078 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-service-ca\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.858226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.858154 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-config\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.859021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.858992 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-service-ca\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.859143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.859076 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-trusted-ca-bundle\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.859484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.859464 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-oauth-serving-cert\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.859869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.859843 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-tmp\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.860672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.859970 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-config\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.860672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.860227 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-serving-cert\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.860672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.860501 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-oauth-config\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.871477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.871456 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ttq2\" (UniqueName: \"kubernetes.io/projected/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-kube-api-access-6ttq2\") pod \"console-5f48f6db46-tjpqf\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:05.953776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:05.953732 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:06.001937 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:06.001324 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50430d1-c382-4a45-9672-d4cb509f8aaf" path="/var/lib/kubelet/pods/c50430d1-c382-4a45-9672-d4cb509f8aaf/volumes" Oct 31 13:05:06.231015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:06.230833 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5f48f6db46-tjpqf"] Oct 31 13:05:06.233711 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:05:06.233686 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50df7b0_3313_46ac_ad9f_a0b8c24fbace.slice/crio-758eb30eb973e83a21c806deb55833a231136408fac6321edc7ae202b38bcd23 WatchSource:0}: Error finding container 758eb30eb973e83a21c806deb55833a231136408fac6321edc7ae202b38bcd23: Status 404 returned error can't find the container with id 758eb30eb973e83a21c806deb55833a231136408fac6321edc7ae202b38bcd23 Oct 31 13:05:06.469333 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:06.469299 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f48f6db46-tjpqf" event={"ID":"c50df7b0-3313-46ac-ad9f-a0b8c24fbace","Type":"ContainerStarted","Data":"4f113937284e7f853723dbe9af7d0e00c009b35c2ec10d2add5793b65ad502d4"} Oct 31 13:05:06.469827 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:06.469356 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f48f6db46-tjpqf" event={"ID":"c50df7b0-3313-46ac-ad9f-a0b8c24fbace","Type":"ContainerStarted","Data":"758eb30eb973e83a21c806deb55833a231136408fac6321edc7ae202b38bcd23"} Oct 31 13:05:06.472780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:06.472750 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e683f34-6fee-4e68-a479-6688ab8221c3","Type":"ContainerStarted","Data":"6fe5f48edbf19b23ff225be9f1b08e75e0f099904da35494297ca22b71b98eb2"} Oct 31 13:05:06.472903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:06.472821 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e683f34-6fee-4e68-a479-6688ab8221c3","Type":"ContainerStarted","Data":"e3ed93f93905f7fe31152bb176002a1ebfd78d6f969bdc4d4cd914754084804e"} Oct 31 13:05:06.472903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:06.472862 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e683f34-6fee-4e68-a479-6688ab8221c3","Type":"ContainerStarted","Data":"9de38637e854de2bc707cd9c80e88f14d3f93f0da5801a7653d0a86c96d398a6"} Oct 31 13:05:06.472903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:06.472899 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e683f34-6fee-4e68-a479-6688ab8221c3","Type":"ContainerStarted","Data":"7c8c211fcbfeac86cb86b9d7114fa000833bad579044487cad79b0de8731bada"} Oct 31 13:05:06.472999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:06.472941 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e683f34-6fee-4e68-a479-6688ab8221c3","Type":"ContainerStarted","Data":"89a47ed69d5f97e65c5d7ccbea07ace9d7a6dcc8c4bfc46d9a0a7f97aa68a2a2"} Oct 31 13:05:06.472999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:06.472980 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"0e683f34-6fee-4e68-a479-6688ab8221c3","Type":"ContainerStarted","Data":"e5fb6e4735d18a300c49833832f666c68e74fc6568b0248f849c0f3ce3ffd058"} Oct 31 13:05:06.492068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:06.492020 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5f48f6db46-tjpqf" podStartSLOduration=1.492004778 podStartE2EDuration="1.492004778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:06.49178676 +0000 UTC m=+74.637530590" watchObservedRunningTime="2026-10-31 13:05:06.492004778 +0000 UTC m=+74.637748614" Oct 31 13:05:06.529814 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:06.529735 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.513448389 podStartE2EDuration="2.529721001s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.272612ms" isStatefulPod=false podCreationTimestamp="2026-10-31 13:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:05:06.52907764 +0000 UTC m=+74.674821468" watchObservedRunningTime="2026-10-31 13:05:06.529721001 +0000 UTC m=+74.675464865" Oct 31 13:05:09.857487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:09.857441 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:05:15.954781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:15.954737 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:15.955151 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:15.954818 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:15.959701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:15.959678 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:16.519404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:16.519375 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:05:16.572107 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:16.572075 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bcd5bb579-ngn5l"] Oct 31 13:05:31.760144 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:05:31.760066 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:05:41.591795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.591752 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6bcd5bb579-ngn5l" podUID="4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" containerName="console" containerID="cri-o://617f5c686619d1dc97780f2b1a717eae1fe721ba382a8cc3a8dbb5d0743faf90" gracePeriod=15 Oct 31 13:05:41.835025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.835000 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:05:41.964250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.964195 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-config\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-config\") pod \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " Oct 31 13:05:41.964250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.964258 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-serving-cert\") pod \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " Oct 31 13:05:41.964480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.964286 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-trusted-ca-bundle\") pod \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " Oct 31 13:05:41.964480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.964318 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-oauth-serving-cert\") pod \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " Oct 31 13:05:41.964480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.964350 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-tmp\") pod \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " Oct 31 13:05:41.964480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.964379 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-kube-api-access-ntqfw\" (UniqueName: \"kubernetes.io/projected/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-kube-api-access-ntqfw\") pod \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " Oct 31 13:05:41.964480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.964434 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-oauth-config\") pod \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " Oct 31 13:05:41.964480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.964458 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-service-ca\") pod \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\" (UID: \"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb\") " Oct 31 13:05:41.964791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.964675 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-tmp" pod "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" (UID: "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:05:41.964791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.964712 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-config" pod "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" (UID: "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:41.964912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.964880 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-trusted-ca-bundle" pod "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" (UID: "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:41.965038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.965004 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-oauth-serving-cert" pod "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" (UID: "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:41.965038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.965022 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-service-ca" pod "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" (UID: "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:05:41.966750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.966718 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-oauth-config" pod "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" (UID: "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:41.966854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.966831 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-serving-cert" pod "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" (UID: "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:05:41.966893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:41.966853 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-kube-api-access-ntqfw" pod "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" (UID: "4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb"). InnerVolumeSpecName "kube-api-access-ntqfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:05:42.065725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.065692 5532 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-tmp\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:42.065725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.065722 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntqfw\" (UniqueName: \"kubernetes.io/projected/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-kube-api-access-ntqfw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:42.065911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.065734 5532 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-oauth-config\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:42.065911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.065745 5532 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:42.065911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.065754 5532 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-config\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:42.065911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.065762 5532 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-console-serving-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:42.065911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.065770 5532 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-trusted-ca-bundle\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:42.065911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.065779 5532 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb-oauth-serving-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:05:42.619579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.619544 5532 generic.go:356] "Generic (PLEG): container finished" podID="4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" containerID="617f5c686619d1dc97780f2b1a717eae1fe721ba382a8cc3a8dbb5d0743faf90" exitCode=0 Oct 31 13:05:42.620027 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.619633 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcd5bb579-ngn5l" event={"ID":"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb","Type":"ContainerDied","Data":"617f5c686619d1dc97780f2b1a717eae1fe721ba382a8cc3a8dbb5d0743faf90"} Oct 31 13:05:42.620027 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.619649 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bcd5bb579-ngn5l" Oct 31 13:05:42.620027 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.619720 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bcd5bb579-ngn5l" event={"ID":"4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb","Type":"ContainerDied","Data":"e91821635399a920ae255ae47410ad66d792e1c7e1bd5a899ba94dbae9e6b10a"} Oct 31 13:05:42.620027 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.619742 5532 scope.go:122] "RemoveContainer" containerID="617f5c686619d1dc97780f2b1a717eae1fe721ba382a8cc3a8dbb5d0743faf90" Oct 31 13:05:42.627860 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.627844 5532 scope.go:122] "RemoveContainer" containerID="617f5c686619d1dc97780f2b1a717eae1fe721ba382a8cc3a8dbb5d0743faf90" Oct 31 13:05:42.628184 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:05:42.628164 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"617f5c686619d1dc97780f2b1a717eae1fe721ba382a8cc3a8dbb5d0743faf90\": container with ID starting with 617f5c686619d1dc97780f2b1a717eae1fe721ba382a8cc3a8dbb5d0743faf90 not found: ID does not exist" containerID="617f5c686619d1dc97780f2b1a717eae1fe721ba382a8cc3a8dbb5d0743faf90" Oct 31 13:05:42.628263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.628195 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"617f5c686619d1dc97780f2b1a717eae1fe721ba382a8cc3a8dbb5d0743faf90"} err="failed to get container status \"617f5c686619d1dc97780f2b1a717eae1fe721ba382a8cc3a8dbb5d0743faf90\": rpc error: code = NotFound desc = could not find container \"617f5c686619d1dc97780f2b1a717eae1fe721ba382a8cc3a8dbb5d0743faf90\": container with ID starting with 617f5c686619d1dc97780f2b1a717eae1fe721ba382a8cc3a8dbb5d0743faf90 not found: ID does not exist" Oct 31 13:05:42.638077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.638056 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6bcd5bb579-ngn5l"] Oct 31 13:05:42.644845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:42.644825 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6bcd5bb579-ngn5l"] Oct 31 13:05:43.999849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:05:43.999816 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" path="/var/lib/kubelet/pods/4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb/volumes" Oct 31 13:06:01.759844 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:06:01.759797 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:06:04.856938 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:04.856905 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:06:04.872647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:04.872617 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:06:05.721235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:05.721210 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 31 13:06:15.813650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.813620 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6bb896f648-h8cht"] Oct 31 13:06:15.814083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.813973 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" containerName="console" Oct 31 13:06:15.814083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.813984 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" containerName="console" Oct 31 13:06:15.814083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.814040 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bfa1c3d-3bb4-42b6-b42f-ac83ef685ddb" containerName="console" Oct 31 13:06:15.816626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.816587 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.839632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.839606 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bb896f648-h8cht"] Oct 31 13:06:15.843966 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.843944 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80cc2424-4b62-4667-b63d-ad7bca01c413-oauth-serving-cert\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.844081 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.843986 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80cc2424-4b62-4667-b63d-ad7bca01c413-console-config\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.844143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.844116 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80cc2424-4b62-4667-b63d-ad7bca01c413-console-serving-cert\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.844196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.844161 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80cc2424-4b62-4667-b63d-ad7bca01c413-trusted-ca-bundle\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.844286 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.844259 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80cc2424-4b62-4667-b63d-ad7bca01c413-tmp\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.844366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.844347 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80cc2424-4b62-4667-b63d-ad7bca01c413-console-oauth-config\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.844414 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.844385 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnljc\" (UniqueName: \"kubernetes.io/projected/80cc2424-4b62-4667-b63d-ad7bca01c413-kube-api-access-hnljc\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.844414 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.844403 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80cc2424-4b62-4667-b63d-ad7bca01c413-service-ca\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.944963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.944929 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80cc2424-4b62-4667-b63d-ad7bca01c413-oauth-serving-cert\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.944963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.944966 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80cc2424-4b62-4667-b63d-ad7bca01c413-console-config\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.945205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.945009 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80cc2424-4b62-4667-b63d-ad7bca01c413-console-serving-cert\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.945205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.945032 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80cc2424-4b62-4667-b63d-ad7bca01c413-trusted-ca-bundle\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.945205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.945049 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80cc2424-4b62-4667-b63d-ad7bca01c413-tmp\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.945205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.945075 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80cc2424-4b62-4667-b63d-ad7bca01c413-console-oauth-config\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.945205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.945093 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnljc\" (UniqueName: \"kubernetes.io/projected/80cc2424-4b62-4667-b63d-ad7bca01c413-kube-api-access-hnljc\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.945205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.945112 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80cc2424-4b62-4667-b63d-ad7bca01c413-service-ca\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.945589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.945503 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80cc2424-4b62-4667-b63d-ad7bca01c413-tmp\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.945790 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.945765 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/80cc2424-4b62-4667-b63d-ad7bca01c413-oauth-serving-cert\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.945877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.945826 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/80cc2424-4b62-4667-b63d-ad7bca01c413-console-config\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.945922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.945907 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/80cc2424-4b62-4667-b63d-ad7bca01c413-service-ca\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.946043 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.946025 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/80cc2424-4b62-4667-b63d-ad7bca01c413-trusted-ca-bundle\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.947433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.947414 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/80cc2424-4b62-4667-b63d-ad7bca01c413-console-serving-cert\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.947742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.947721 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/80cc2424-4b62-4667-b63d-ad7bca01c413-console-oauth-config\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:15.958065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:15.958045 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnljc\" (UniqueName: \"kubernetes.io/projected/80cc2424-4b62-4667-b63d-ad7bca01c413-kube-api-access-hnljc\") pod \"console-6bb896f648-h8cht\" (UID: \"80cc2424-4b62-4667-b63d-ad7bca01c413\") " pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:16.126403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:16.126329 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:16.401616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:16.401569 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6bb896f648-h8cht"] Oct 31 13:06:16.404567 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:06:16.404534 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cc2424_4b62_4667_b63d_ad7bca01c413.slice/crio-d2ab31687d50b59026b26d5d65c8cb8bffa1868a53cb396d19ac4b9a3f7c60c3 WatchSource:0}: Error finding container d2ab31687d50b59026b26d5d65c8cb8bffa1868a53cb396d19ac4b9a3f7c60c3: Status 404 returned error can't find the container with id d2ab31687d50b59026b26d5d65c8cb8bffa1868a53cb396d19ac4b9a3f7c60c3 Oct 31 13:06:16.748537 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:16.748436 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb896f648-h8cht" event={"ID":"80cc2424-4b62-4667-b63d-ad7bca01c413","Type":"ContainerStarted","Data":"de1c27fbed4b9df3071e6ac13ebefcd64072e0c45c81d73714e14235a1f89cc8"} Oct 31 13:06:16.748537 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:16.748504 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6bb896f648-h8cht" event={"ID":"80cc2424-4b62-4667-b63d-ad7bca01c413","Type":"ContainerStarted","Data":"d2ab31687d50b59026b26d5d65c8cb8bffa1868a53cb396d19ac4b9a3f7c60c3"} Oct 31 13:06:16.771941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:16.771900 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6bb896f648-h8cht" podStartSLOduration=1.771888015 podStartE2EDuration="1.771888015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:06:16.771772865 +0000 UTC m=+144.917516693" watchObservedRunningTime="2026-10-31 13:06:16.771888015 +0000 UTC m=+144.917631835" Oct 31 13:06:26.126633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:26.126571 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:26.127044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:26.126650 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:26.131293 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:26.131270 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:26.793268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:26.793239 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6bb896f648-h8cht" Oct 31 13:06:26.846735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:26.846704 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f48f6db46-tjpqf"] Oct 31 13:06:31.760286 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:06:31.760247 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:06:51.867721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:51.867653 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5f48f6db46-tjpqf" podUID="c50df7b0-3313-46ac-ad9f-a0b8c24fbace" containerName="console" containerID="cri-o://4f113937284e7f853723dbe9af7d0e00c009b35c2ec10d2add5793b65ad502d4" gracePeriod=15 Oct 31 13:06:52.116160 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.116133 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:06:52.253085 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.253047 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-trusted-ca-bundle\") pod \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " Oct 31 13:06:52.253294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.253098 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-oauth-serving-cert\") pod \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " Oct 31 13:06:52.253294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.253129 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-config\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-config\") pod \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " Oct 31 13:06:52.253294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.253164 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-tmp\" (UniqueName: \"kubernetes.io/empty-dir/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-tmp\") pod \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " Oct 31 13:06:52.253294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.253191 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-kube-api-access-6ttq2\" (UniqueName: \"kubernetes.io/projected/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-kube-api-access-6ttq2\") pod \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " Oct 31 13:06:52.253294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.253217 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-oauth-config\") pod \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " Oct 31 13:06:52.253294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.253257 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-service-ca\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-service-ca\") pod \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " Oct 31 13:06:52.253589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.253317 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-serving-cert\") pod \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\" (UID: \"c50df7b0-3313-46ac-ad9f-a0b8c24fbace\") " Oct 31 13:06:52.253589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.253533 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-tmp" pod "c50df7b0-3313-46ac-ad9f-a0b8c24fbace" (UID: "c50df7b0-3313-46ac-ad9f-a0b8c24fbace"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:06:52.253732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.253633 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-trusted-ca-bundle" pod "c50df7b0-3313-46ac-ad9f-a0b8c24fbace" (UID: "c50df7b0-3313-46ac-ad9f-a0b8c24fbace"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:52.253732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.253653 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-config" pod "c50df7b0-3313-46ac-ad9f-a0b8c24fbace" (UID: "c50df7b0-3313-46ac-ad9f-a0b8c24fbace"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:52.253732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.253697 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-oauth-serving-cert" pod "c50df7b0-3313-46ac-ad9f-a0b8c24fbace" (UID: "c50df7b0-3313-46ac-ad9f-a0b8c24fbace"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:52.254041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.254012 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-service-ca" pod "c50df7b0-3313-46ac-ad9f-a0b8c24fbace" (UID: "c50df7b0-3313-46ac-ad9f-a0b8c24fbace"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:06:52.255575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.255551 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-serving-cert" pod "c50df7b0-3313-46ac-ad9f-a0b8c24fbace" (UID: "c50df7b0-3313-46ac-ad9f-a0b8c24fbace"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:52.255687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.255607 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-oauth-config" pod "c50df7b0-3313-46ac-ad9f-a0b8c24fbace" (UID: "c50df7b0-3313-46ac-ad9f-a0b8c24fbace"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:06:52.255737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.255723 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-kube-api-access-6ttq2" pod "c50df7b0-3313-46ac-ad9f-a0b8c24fbace" (UID: "c50df7b0-3313-46ac-ad9f-a0b8c24fbace"). InnerVolumeSpecName "kube-api-access-6ttq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:06:52.354750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.354715 5532 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:52.354750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.354744 5532 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-serving-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:52.354750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.354754 5532 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-trusted-ca-bundle\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:52.354980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.354762 5532 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-oauth-serving-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:52.354980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.354771 5532 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-config\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:52.354980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.354781 5532 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-tmp\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:52.354980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.354789 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ttq2\" (UniqueName: \"kubernetes.io/projected/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-kube-api-access-6ttq2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:52.354980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.354798 5532 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c50df7b0-3313-46ac-ad9f-a0b8c24fbace-console-oauth-config\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:06:52.888342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.888307 5532 generic.go:356] "Generic (PLEG): container finished" podID="c50df7b0-3313-46ac-ad9f-a0b8c24fbace" containerID="4f113937284e7f853723dbe9af7d0e00c009b35c2ec10d2add5793b65ad502d4" exitCode=0 Oct 31 13:06:52.888803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.888352 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f48f6db46-tjpqf" event={"ID":"c50df7b0-3313-46ac-ad9f-a0b8c24fbace","Type":"ContainerDied","Data":"4f113937284e7f853723dbe9af7d0e00c009b35c2ec10d2add5793b65ad502d4"} Oct 31 13:06:52.888803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.888368 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5f48f6db46-tjpqf" Oct 31 13:06:52.888803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.888425 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5f48f6db46-tjpqf" event={"ID":"c50df7b0-3313-46ac-ad9f-a0b8c24fbace","Type":"ContainerDied","Data":"758eb30eb973e83a21c806deb55833a231136408fac6321edc7ae202b38bcd23"} Oct 31 13:06:52.888803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.888447 5532 scope.go:122] "RemoveContainer" containerID="4f113937284e7f853723dbe9af7d0e00c009b35c2ec10d2add5793b65ad502d4" Oct 31 13:06:52.897716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.897699 5532 scope.go:122] "RemoveContainer" containerID="4f113937284e7f853723dbe9af7d0e00c009b35c2ec10d2add5793b65ad502d4" Oct 31 13:06:52.897993 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:06:52.897970 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f113937284e7f853723dbe9af7d0e00c009b35c2ec10d2add5793b65ad502d4\": container with ID starting with 4f113937284e7f853723dbe9af7d0e00c009b35c2ec10d2add5793b65ad502d4 not found: ID does not exist" containerID="4f113937284e7f853723dbe9af7d0e00c009b35c2ec10d2add5793b65ad502d4" Oct 31 13:06:52.898046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.898002 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f113937284e7f853723dbe9af7d0e00c009b35c2ec10d2add5793b65ad502d4"} err="failed to get container status \"4f113937284e7f853723dbe9af7d0e00c009b35c2ec10d2add5793b65ad502d4\": rpc error: code = NotFound desc = could not find container \"4f113937284e7f853723dbe9af7d0e00c009b35c2ec10d2add5793b65ad502d4\": container with ID starting with 4f113937284e7f853723dbe9af7d0e00c009b35c2ec10d2add5793b65ad502d4 not found: ID does not exist" Oct 31 13:06:52.912044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.912020 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5f48f6db46-tjpqf"] Oct 31 13:06:52.916653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:52.916569 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5f48f6db46-tjpqf"] Oct 31 13:06:54.000892 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:06:54.000858 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50df7b0-3313-46ac-ad9f-a0b8c24fbace" path="/var/lib/kubelet/pods/c50df7b0-3313-46ac-ad9f-a0b8c24fbace/volumes" Oct 31 13:07:01.759677 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:07:01.759584 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:07:31.759543 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:07:31.759504 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:08:01.759543 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:08:01.759497 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:08:31.760536 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:08:31.760460 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:09:01.760161 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:09:01.760122 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:09:31.760407 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:09:31.760366 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:09:42.700735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.700702 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4"] Oct 31 13:09:42.701178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.701021 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c50df7b0-3313-46ac-ad9f-a0b8c24fbace" containerName="console" Oct 31 13:09:42.701178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.701031 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50df7b0-3313-46ac-ad9f-a0b8c24fbace" containerName="console" Oct 31 13:09:42.701178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.701097 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50df7b0-3313-46ac-ad9f-a0b8c24fbace" containerName="console" Oct 31 13:09:42.703939 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.703922 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4" Oct 31 13:09:42.710639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.710613 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-d269c\"" Oct 31 13:09:42.710779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.710668 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 31 13:09:42.710779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.710696 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 31 13:09:42.717098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.717078 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4"] Oct 31 13:09:42.772201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.772168 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpzlb\" (UniqueName: \"kubernetes.io/projected/1e4301d5-d442-499b-88ad-1da7e056f55f-kube-api-access-wpzlb\") pod \"jobset-operator-674cd59dc5-hrxs4\" (UID: \"1e4301d5-d442-499b-88ad-1da7e056f55f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4" Oct 31 13:09:42.772359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.772218 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e4301d5-d442-499b-88ad-1da7e056f55f-tmp\") pod \"jobset-operator-674cd59dc5-hrxs4\" (UID: \"1e4301d5-d442-499b-88ad-1da7e056f55f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4" Oct 31 13:09:42.873261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.873223 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpzlb\" (UniqueName: \"kubernetes.io/projected/1e4301d5-d442-499b-88ad-1da7e056f55f-kube-api-access-wpzlb\") pod \"jobset-operator-674cd59dc5-hrxs4\" (UID: \"1e4301d5-d442-499b-88ad-1da7e056f55f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4" Oct 31 13:09:42.873416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.873273 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e4301d5-d442-499b-88ad-1da7e056f55f-tmp\") pod \"jobset-operator-674cd59dc5-hrxs4\" (UID: \"1e4301d5-d442-499b-88ad-1da7e056f55f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4" Oct 31 13:09:42.873622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.873586 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e4301d5-d442-499b-88ad-1da7e056f55f-tmp\") pod \"jobset-operator-674cd59dc5-hrxs4\" (UID: \"1e4301d5-d442-499b-88ad-1da7e056f55f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4" Oct 31 13:09:42.891912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:42.891881 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpzlb\" (UniqueName: \"kubernetes.io/projected/1e4301d5-d442-499b-88ad-1da7e056f55f-kube-api-access-wpzlb\") pod \"jobset-operator-674cd59dc5-hrxs4\" (UID: \"1e4301d5-d442-499b-88ad-1da7e056f55f\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4" Oct 31 13:09:43.023785 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:43.023687 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4" Oct 31 13:09:43.297021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:43.296943 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4"] Oct 31 13:09:43.299963 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:09:43.299936 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e4301d5_d442_499b_88ad_1da7e056f55f.slice/crio-53ff0ed048a5662d44a227adc9d1b2d571f3780e804d081b3f1929f8e3117015 WatchSource:0}: Error finding container 53ff0ed048a5662d44a227adc9d1b2d571f3780e804d081b3f1929f8e3117015: Status 404 returned error can't find the container with id 53ff0ed048a5662d44a227adc9d1b2d571f3780e804d081b3f1929f8e3117015 Oct 31 13:09:43.301566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:43.301547 5532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:09:43.527013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:43.526984 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4" event={"ID":"1e4301d5-d442-499b-88ad-1da7e056f55f","Type":"ContainerStarted","Data":"53ff0ed048a5662d44a227adc9d1b2d571f3780e804d081b3f1929f8e3117015"} Oct 31 13:09:46.543060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:46.543023 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4" event={"ID":"1e4301d5-d442-499b-88ad-1da7e056f55f","Type":"ContainerStarted","Data":"ebfe0fbd969fad2917ed44ec44b8c455c7c2508f242b3627748b43c102f8961b"} Oct 31 13:09:46.564107 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:46.564058 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-hrxs4" podStartSLOduration=1.736931737 podStartE2EDuration="4.564045665s" totalImagesPullingTime="2.827113928s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:46.563758647 +0000 UTC m=+354.709502475" watchObservedRunningTime="2026-10-31 13:09:46.564045665 +0000 UTC m=+354.709789492" Oct 31 13:09:47.587832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.587794 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls"] Oct 31 13:09:47.591547 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.591524 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.612428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.612399 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/312b3881-e12a-441d-b3de-bd68618f0db8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.612570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.612432 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/312b3881-e12a-441d-b3de-bd68618f0db8-tmp\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.612570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.612451 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/312b3881-e12a-441d-b3de-bd68618f0db8-manifests\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.612570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.612550 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/312b3881-e12a-441d-b3de-bd68618f0db8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.612718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.612635 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m97x4\" (UniqueName: \"kubernetes.io/projected/312b3881-e12a-441d-b3de-bd68618f0db8-kube-api-access-m97x4\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.612718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.612656 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/312b3881-e12a-441d-b3de-bd68618f0db8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.621581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.621554 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls"] Oct 31 13:09:47.663556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.663526 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 31 13:09:47.663762 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.663734 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 31 13:09:47.663825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.663758 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 31 13:09:47.663825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.663796 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-jj88f\"" Oct 31 13:09:47.665110 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.665093 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 31 13:09:47.665341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.665325 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 31 13:09:47.713880 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.713842 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/312b3881-e12a-441d-b3de-bd68618f0db8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.714050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.713882 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/312b3881-e12a-441d-b3de-bd68618f0db8-tmp\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.714050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.713912 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/312b3881-e12a-441d-b3de-bd68618f0db8-manifests\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.714050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.713941 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/312b3881-e12a-441d-b3de-bd68618f0db8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.714259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.714117 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m97x4\" (UniqueName: \"kubernetes.io/projected/312b3881-e12a-441d-b3de-bd68618f0db8-kube-api-access-m97x4\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.714259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.714161 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/312b3881-e12a-441d-b3de-bd68618f0db8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.714464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.714432 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/312b3881-e12a-441d-b3de-bd68618f0db8-manifests\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.716333 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.716310 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/312b3881-e12a-441d-b3de-bd68618f0db8-tmp\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.716434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.716402 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/312b3881-e12a-441d-b3de-bd68618f0db8-metrics-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.716729 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.716706 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/312b3881-e12a-441d-b3de-bd68618f0db8-webhook-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.716729 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.716697 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/312b3881-e12a-441d-b3de-bd68618f0db8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.727327 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.727303 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m97x4\" (UniqueName: \"kubernetes.io/projected/312b3881-e12a-441d-b3de-bd68618f0db8-kube-api-access-m97x4\") pod \"opendatahub-operator-controller-manager-7748d49f5-zm9ls\" (UID: \"312b3881-e12a-441d-b3de-bd68618f0db8\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:47.906329 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:47.906298 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:48.209867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:48.209835 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls"] Oct 31 13:09:48.210837 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:09:48.210809 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod312b3881_e12a_441d_b3de_bd68618f0db8.slice/crio-b810d41f860916f0988f80c4cb68382ab5a8a69952b7270077047f990142c1e1 WatchSource:0}: Error finding container b810d41f860916f0988f80c4cb68382ab5a8a69952b7270077047f990142c1e1: Status 404 returned error can't find the container with id b810d41f860916f0988f80c4cb68382ab5a8a69952b7270077047f990142c1e1 Oct 31 13:09:48.553268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:48.553185 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" event={"ID":"312b3881-e12a-441d-b3de-bd68618f0db8","Type":"ContainerStarted","Data":"b810d41f860916f0988f80c4cb68382ab5a8a69952b7270077047f990142c1e1"} Oct 31 13:09:51.567121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:51.567086 5532 generic.go:356] "Generic (PLEG): container finished" podID="312b3881-e12a-441d-b3de-bd68618f0db8" containerID="875fd7b850ba79ebecbf4bdd583790a2eafa518663c521ed445e4bc45b669adf" exitCode=0 Oct 31 13:09:51.567486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:51.567143 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" event={"ID":"312b3881-e12a-441d-b3de-bd68618f0db8","Type":"ContainerDied","Data":"875fd7b850ba79ebecbf4bdd583790a2eafa518663c521ed445e4bc45b669adf"} Oct 31 13:09:52.573397 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:52.573364 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" event={"ID":"312b3881-e12a-441d-b3de-bd68618f0db8","Type":"ContainerStarted","Data":"68457dfb7e60a94a9f70dff561b79ccade8741cf8cff9a44eb6b90385e3e8e8d"} Oct 31 13:09:52.573784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:52.573469 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:09:52.642060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:09:52.642003 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" podStartSLOduration=1.965057168 podStartE2EDuration="5.641987991s" totalImagesPullingTime="3.247046398s" totalInitContainerRuntime="429.884425ms" isStatefulPod=false podCreationTimestamp="2026-10-31 13:09:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:09:52.640109604 +0000 UTC m=+360.785853431" watchObservedRunningTime="2026-10-31 13:09:52.641987991 +0000 UTC m=+360.787731818" Oct 31 13:10:01.760677 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:10:01.760577 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:10:03.580473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:03.580436 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-7748d49f5-zm9ls" Oct 31 13:10:09.053670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.053633 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm"] Oct 31 13:10:09.059382 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.059362 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm" Oct 31 13:10:09.068677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.068653 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm"] Oct 31 13:10:09.069212 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.069193 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 31 13:10:09.069283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.069215 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 31 13:10:09.095529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.095507 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8cwth\"" Oct 31 13:10:09.202861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.202823 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6stdl\" (UniqueName: \"kubernetes.io/projected/2c674c90-2b7a-4787-9afe-149a282ae967-kube-api-access-6stdl\") pod \"cert-manager-cainjector-5c57bbd4fc-tvfmm\" (UID: \"2c674c90-2b7a-4787-9afe-149a282ae967\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm" Oct 31 13:10:09.202861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.202862 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c674c90-2b7a-4787-9afe-149a282ae967-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tvfmm\" (UID: \"2c674c90-2b7a-4787-9afe-149a282ae967\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm" Oct 31 13:10:09.303665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.303630 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6stdl\" (UniqueName: \"kubernetes.io/projected/2c674c90-2b7a-4787-9afe-149a282ae967-kube-api-access-6stdl\") pod \"cert-manager-cainjector-5c57bbd4fc-tvfmm\" (UID: \"2c674c90-2b7a-4787-9afe-149a282ae967\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm" Oct 31 13:10:09.303835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.303675 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c674c90-2b7a-4787-9afe-149a282ae967-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tvfmm\" (UID: \"2c674c90-2b7a-4787-9afe-149a282ae967\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm" Oct 31 13:10:09.325163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.325129 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2c674c90-2b7a-4787-9afe-149a282ae967-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tvfmm\" (UID: \"2c674c90-2b7a-4787-9afe-149a282ae967\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm" Oct 31 13:10:09.325163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.325153 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6stdl\" (UniqueName: \"kubernetes.io/projected/2c674c90-2b7a-4787-9afe-149a282ae967-kube-api-access-6stdl\") pod \"cert-manager-cainjector-5c57bbd4fc-tvfmm\" (UID: \"2c674c90-2b7a-4787-9afe-149a282ae967\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm" Oct 31 13:10:09.371476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.371450 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm" Oct 31 13:10:09.642812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:09.642744 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm"] Oct 31 13:10:09.646065 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:10:09.646027 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2c674c90_2b7a_4787_9afe_149a282ae967.slice/crio-65b6beb2a7656d192cbb63eee1a5c6eaa750db3446efcdfd5681cbdd4d634d21 WatchSource:0}: Error finding container 65b6beb2a7656d192cbb63eee1a5c6eaa750db3446efcdfd5681cbdd4d634d21: Status 404 returned error can't find the container with id 65b6beb2a7656d192cbb63eee1a5c6eaa750db3446efcdfd5681cbdd4d634d21 Oct 31 13:10:10.646931 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:10.646890 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm" event={"ID":"2c674c90-2b7a-4787-9afe-149a282ae967","Type":"ContainerStarted","Data":"65b6beb2a7656d192cbb63eee1a5c6eaa750db3446efcdfd5681cbdd4d634d21"} Oct 31 13:10:13.652422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:13.652390 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw"] Oct 31 13:10:13.654542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:13.654522 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" Oct 31 13:10:13.662565 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:13.662540 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm" event={"ID":"2c674c90-2b7a-4787-9afe-149a282ae967","Type":"ContainerStarted","Data":"7e40be604a0f724d422e3a2f52eeb7d5d568c20063f09b9c90829444b1a39e20"} Oct 31 13:10:13.663127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:13.663105 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-stlsn\"" Oct 31 13:10:13.667872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:13.667845 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw"] Oct 31 13:10:13.703607 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:13.703539 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tvfmm" podStartSLOduration=1.2911258380000001 podStartE2EDuration="4.703525465s" totalImagesPullingTime="3.412399627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:13.702780089 +0000 UTC m=+381.848523922" watchObservedRunningTime="2026-10-31 13:10:13.703525465 +0000 UTC m=+381.849269292" Oct 31 13:10:13.743514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:13.743477 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51b75119-a5cd-4e8c-bd06-3725a6bb30a5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-48lfw\" (UID: \"51b75119-a5cd-4e8c-bd06-3725a6bb30a5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" Oct 31 13:10:13.743676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:13.743526 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ln5f\" (UniqueName: \"kubernetes.io/projected/51b75119-a5cd-4e8c-bd06-3725a6bb30a5-kube-api-access-2ln5f\") pod \"cert-manager-webhook-5dbbbf75cc-48lfw\" (UID: \"51b75119-a5cd-4e8c-bd06-3725a6bb30a5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" Oct 31 13:10:13.844790 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:13.844750 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51b75119-a5cd-4e8c-bd06-3725a6bb30a5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-48lfw\" (UID: \"51b75119-a5cd-4e8c-bd06-3725a6bb30a5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" Oct 31 13:10:13.844790 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:13.844791 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ln5f\" (UniqueName: \"kubernetes.io/projected/51b75119-a5cd-4e8c-bd06-3725a6bb30a5-kube-api-access-2ln5f\") pod \"cert-manager-webhook-5dbbbf75cc-48lfw\" (UID: \"51b75119-a5cd-4e8c-bd06-3725a6bb30a5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" Oct 31 13:10:13.858394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:13.858366 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/51b75119-a5cd-4e8c-bd06-3725a6bb30a5-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-48lfw\" (UID: \"51b75119-a5cd-4e8c-bd06-3725a6bb30a5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" Oct 31 13:10:13.858515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:13.858420 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ln5f\" (UniqueName: \"kubernetes.io/projected/51b75119-a5cd-4e8c-bd06-3725a6bb30a5-kube-api-access-2ln5f\") pod \"cert-manager-webhook-5dbbbf75cc-48lfw\" (UID: \"51b75119-a5cd-4e8c-bd06-3725a6bb30a5\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" Oct 31 13:10:13.965867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:13.965784 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" Oct 31 13:10:14.238118 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:14.238035 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw"] Oct 31 13:10:14.240922 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:10:14.240895 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51b75119_a5cd_4e8c_bd06_3725a6bb30a5.slice/crio-38d3dc97112abab1688a9374f1c2f541afebf45d2d11155a91e3f1c57139f8b4 WatchSource:0}: Error finding container 38d3dc97112abab1688a9374f1c2f541afebf45d2d11155a91e3f1c57139f8b4: Status 404 returned error can't find the container with id 38d3dc97112abab1688a9374f1c2f541afebf45d2d11155a91e3f1c57139f8b4 Oct 31 13:10:14.668657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:14.668616 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" event={"ID":"51b75119-a5cd-4e8c-bd06-3725a6bb30a5","Type":"ContainerStarted","Data":"97d720d344ba2de5bc468237221f3476baf6536594a5916f5cc9d163bec058e4"} Oct 31 13:10:14.669032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:14.668871 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" Oct 31 13:10:14.669032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:14.668898 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" event={"ID":"51b75119-a5cd-4e8c-bd06-3725a6bb30a5","Type":"ContainerStarted","Data":"38d3dc97112abab1688a9374f1c2f541afebf45d2d11155a91e3f1c57139f8b4"} Oct 31 13:10:14.690759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:14.690714 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" podStartSLOduration=1.690702206 podStartE2EDuration="1.690702206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:14.690105533 +0000 UTC m=+382.835849362" watchObservedRunningTime="2026-10-31 13:10:14.690702206 +0000 UTC m=+382.836446034" Oct 31 13:10:15.885551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:15.885517 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-b7f5f"] Oct 31 13:10:15.891885 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:15.888964 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-b7f5f" Oct 31 13:10:15.897611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:15.897566 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-dg94j\"" Oct 31 13:10:15.898207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:15.898182 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-b7f5f"] Oct 31 13:10:15.965736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:15.965702 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47nkp\" (UniqueName: \"kubernetes.io/projected/c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2-kube-api-access-47nkp\") pod \"cert-manager-597b95cb84-b7f5f\" (UID: \"c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2\") " pod="cert-manager/cert-manager-597b95cb84-b7f5f" Oct 31 13:10:15.965900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:15.965767 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2-bound-sa-token\") pod \"cert-manager-597b95cb84-b7f5f\" (UID: \"c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2\") " pod="cert-manager/cert-manager-597b95cb84-b7f5f" Oct 31 13:10:16.067063 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:16.067019 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2-bound-sa-token\") pod \"cert-manager-597b95cb84-b7f5f\" (UID: \"c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2\") " pod="cert-manager/cert-manager-597b95cb84-b7f5f" Oct 31 13:10:16.067257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:16.067113 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47nkp\" (UniqueName: \"kubernetes.io/projected/c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2-kube-api-access-47nkp\") pod \"cert-manager-597b95cb84-b7f5f\" (UID: \"c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2\") " pod="cert-manager/cert-manager-597b95cb84-b7f5f" Oct 31 13:10:16.080742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:16.080701 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2-bound-sa-token\") pod \"cert-manager-597b95cb84-b7f5f\" (UID: \"c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2\") " pod="cert-manager/cert-manager-597b95cb84-b7f5f" Oct 31 13:10:16.080861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:16.080812 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47nkp\" (UniqueName: \"kubernetes.io/projected/c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2-kube-api-access-47nkp\") pod \"cert-manager-597b95cb84-b7f5f\" (UID: \"c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2\") " pod="cert-manager/cert-manager-597b95cb84-b7f5f" Oct 31 13:10:16.203291 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:16.203255 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-b7f5f" Oct 31 13:10:16.476145 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:16.476072 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-b7f5f"] Oct 31 13:10:16.478854 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:10:16.478822 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0275f2e_d2ac_426b_b45d_bb94ec8a8dc2.slice/crio-b8ea513f5e8762d4289c45808d38555546911ad44aa2037b3f5b0d786e0be81e WatchSource:0}: Error finding container b8ea513f5e8762d4289c45808d38555546911ad44aa2037b3f5b0d786e0be81e: Status 404 returned error can't find the container with id b8ea513f5e8762d4289c45808d38555546911ad44aa2037b3f5b0d786e0be81e Oct 31 13:10:16.682090 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:16.682055 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-b7f5f" event={"ID":"c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2","Type":"ContainerStarted","Data":"8ef52b6b32f2cc22fd1a7e4d09eb365ecd3ddfcff3a5f339f5135becd3d72022"} Oct 31 13:10:16.682090 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:16.682096 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-b7f5f" event={"ID":"c0275f2e-d2ac-426b-b45d-bb94ec8a8dc2","Type":"ContainerStarted","Data":"b8ea513f5e8762d4289c45808d38555546911ad44aa2037b3f5b0d786e0be81e"} Oct 31 13:10:16.712143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:16.712092 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-b7f5f" podStartSLOduration=1.712079194 podStartE2EDuration="1.712079194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:16.710723748 +0000 UTC m=+384.856467577" watchObservedRunningTime="2026-10-31 13:10:16.712079194 +0000 UTC m=+384.857823022" Oct 31 13:10:18.811841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.811808 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7485898565-kp66f"] Oct 31 13:10:18.814444 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.814418 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" Oct 31 13:10:18.823444 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.823420 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 31 13:10:18.824239 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.824209 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 31 13:10:18.831016 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.830993 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7485898565-kp66f"] Oct 31 13:10:18.890114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.890079 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 31 13:10:18.890114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.890088 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 31 13:10:18.890452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.890433 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-6nszp\"" Oct 31 13:10:18.891928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.891899 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87eda91-1ec7-42bf-bf88-250b804bd332-tls-certs\") pod \"kube-auth-proxy-7485898565-kp66f\" (UID: \"c87eda91-1ec7-42bf-bf88-250b804bd332\") " pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" Oct 31 13:10:18.892067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.892015 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c87eda91-1ec7-42bf-bf88-250b804bd332-tmp\") pod \"kube-auth-proxy-7485898565-kp66f\" (UID: \"c87eda91-1ec7-42bf-bf88-250b804bd332\") " pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" Oct 31 13:10:18.892123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.892079 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhncd\" (UniqueName: \"kubernetes.io/projected/c87eda91-1ec7-42bf-bf88-250b804bd332-kube-api-access-xhncd\") pod \"kube-auth-proxy-7485898565-kp66f\" (UID: \"c87eda91-1ec7-42bf-bf88-250b804bd332\") " pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" Oct 31 13:10:18.992773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.992726 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87eda91-1ec7-42bf-bf88-250b804bd332-tls-certs\") pod \"kube-auth-proxy-7485898565-kp66f\" (UID: \"c87eda91-1ec7-42bf-bf88-250b804bd332\") " pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" Oct 31 13:10:18.992968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.992822 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c87eda91-1ec7-42bf-bf88-250b804bd332-tmp\") pod \"kube-auth-proxy-7485898565-kp66f\" (UID: \"c87eda91-1ec7-42bf-bf88-250b804bd332\") " pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" Oct 31 13:10:18.992968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.992853 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhncd\" (UniqueName: \"kubernetes.io/projected/c87eda91-1ec7-42bf-bf88-250b804bd332-kube-api-access-xhncd\") pod \"kube-auth-proxy-7485898565-kp66f\" (UID: \"c87eda91-1ec7-42bf-bf88-250b804bd332\") " pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" Oct 31 13:10:18.996328 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.996269 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c87eda91-1ec7-42bf-bf88-250b804bd332-tmp\") pod \"kube-auth-proxy-7485898565-kp66f\" (UID: \"c87eda91-1ec7-42bf-bf88-250b804bd332\") " pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" Oct 31 13:10:18.996579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:18.996554 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87eda91-1ec7-42bf-bf88-250b804bd332-tls-certs\") pod \"kube-auth-proxy-7485898565-kp66f\" (UID: \"c87eda91-1ec7-42bf-bf88-250b804bd332\") " pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" Oct 31 13:10:19.012604 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:19.012565 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhncd\" (UniqueName: \"kubernetes.io/projected/c87eda91-1ec7-42bf-bf88-250b804bd332-kube-api-access-xhncd\") pod \"kube-auth-proxy-7485898565-kp66f\" (UID: \"c87eda91-1ec7-42bf-bf88-250b804bd332\") " pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" Oct 31 13:10:19.127233 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:19.127161 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" Oct 31 13:10:19.404108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:19.404075 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7485898565-kp66f"] Oct 31 13:10:19.407990 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:10:19.407961 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87eda91_1ec7_42bf_bf88_250b804bd332.slice/crio-c1eb91ff9f080ce47e518c442792fa5cdfeca6a04566c52671b10a6aeae652b6 WatchSource:0}: Error finding container c1eb91ff9f080ce47e518c442792fa5cdfeca6a04566c52671b10a6aeae652b6: Status 404 returned error can't find the container with id c1eb91ff9f080ce47e518c442792fa5cdfeca6a04566c52671b10a6aeae652b6 Oct 31 13:10:19.695856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:19.695778 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" event={"ID":"c87eda91-1ec7-42bf-bf88-250b804bd332","Type":"ContainerStarted","Data":"c1eb91ff9f080ce47e518c442792fa5cdfeca6a04566c52671b10a6aeae652b6"} Oct 31 13:10:20.679114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:20.679083 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" Oct 31 13:10:22.712690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:22.712650 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" event={"ID":"c87eda91-1ec7-42bf-bf88-250b804bd332","Type":"ContainerStarted","Data":"eb02bc52299a46bc605ed25e973f089d444d1b267073e3600b7f513253c542e2"} Oct 31 13:10:22.735049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:22.734995 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7485898565-kp66f" podStartSLOduration=2.296891747 podStartE2EDuration="4.734977433s" totalImagesPullingTime="2.438085686s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:22.733906715 +0000 UTC m=+390.879650544" watchObservedRunningTime="2026-10-31 13:10:22.734977433 +0000 UTC m=+390.880721262" Oct 31 13:10:31.760236 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:10:31.760197 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:10:35.892151 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:35.892115 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2"] Oct 31 13:10:35.895733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:35.895715 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:35.903731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:35.903709 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 31 13:10:35.903831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:35.903754 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 31 13:10:35.903831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:35.903763 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-6svpw\"" Oct 31 13:10:35.907496 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:35.907465 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2"] Oct 31 13:10:35.911486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:35.911465 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 31 13:10:35.945856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:35.945830 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1fe05f4f-3a03-41d7-98e2-abe4c0b13bef-manager-config\") pod \"jobset-controller-manager-7556c48c69-qqdn2\" (UID: \"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef\") " pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:35.945988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:35.945900 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fe05f4f-3a03-41d7-98e2-abe4c0b13bef-metrics-certs\") pod \"jobset-controller-manager-7556c48c69-qqdn2\" (UID: \"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef\") " pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:35.945988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:35.945973 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdl5g\" (UniqueName: \"kubernetes.io/projected/1fe05f4f-3a03-41d7-98e2-abe4c0b13bef-kube-api-access-pdl5g\") pod \"jobset-controller-manager-7556c48c69-qqdn2\" (UID: \"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef\") " pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:35.946067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:35.946014 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fe05f4f-3a03-41d7-98e2-abe4c0b13bef-cert\") pod \"jobset-controller-manager-7556c48c69-qqdn2\" (UID: \"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef\") " pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:36.047312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:36.047274 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fe05f4f-3a03-41d7-98e2-abe4c0b13bef-metrics-certs\") pod \"jobset-controller-manager-7556c48c69-qqdn2\" (UID: \"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef\") " pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:36.047493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:36.047328 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdl5g\" (UniqueName: \"kubernetes.io/projected/1fe05f4f-3a03-41d7-98e2-abe4c0b13bef-kube-api-access-pdl5g\") pod \"jobset-controller-manager-7556c48c69-qqdn2\" (UID: \"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef\") " pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:36.047493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:36.047377 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fe05f4f-3a03-41d7-98e2-abe4c0b13bef-cert\") pod \"jobset-controller-manager-7556c48c69-qqdn2\" (UID: \"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef\") " pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:36.047644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:36.047504 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1fe05f4f-3a03-41d7-98e2-abe4c0b13bef-manager-config\") pod \"jobset-controller-manager-7556c48c69-qqdn2\" (UID: \"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef\") " pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:36.048387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:36.048356 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/1fe05f4f-3a03-41d7-98e2-abe4c0b13bef-manager-config\") pod \"jobset-controller-manager-7556c48c69-qqdn2\" (UID: \"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef\") " pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:36.049844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:36.049819 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1fe05f4f-3a03-41d7-98e2-abe4c0b13bef-metrics-certs\") pod \"jobset-controller-manager-7556c48c69-qqdn2\" (UID: \"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef\") " pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:36.050036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:36.050013 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1fe05f4f-3a03-41d7-98e2-abe4c0b13bef-cert\") pod \"jobset-controller-manager-7556c48c69-qqdn2\" (UID: \"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef\") " pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:36.060464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:36.060436 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdl5g\" (UniqueName: \"kubernetes.io/projected/1fe05f4f-3a03-41d7-98e2-abe4c0b13bef-kube-api-access-pdl5g\") pod \"jobset-controller-manager-7556c48c69-qqdn2\" (UID: \"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef\") " pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:36.208270 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:36.208237 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:36.480847 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:36.480771 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2"] Oct 31 13:10:36.483623 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:10:36.483581 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fe05f4f_3a03_41d7_98e2_abe4c0b13bef.slice/crio-9597ba9802e532829929971979c2de70e0ec0aa6a76ecf958d10c5016d70477c WatchSource:0}: Error finding container 9597ba9802e532829929971979c2de70e0ec0aa6a76ecf958d10c5016d70477c: Status 404 returned error can't find the container with id 9597ba9802e532829929971979c2de70e0ec0aa6a76ecf958d10c5016d70477c Oct 31 13:10:36.772192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:36.772110 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" event={"ID":"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef","Type":"ContainerStarted","Data":"9597ba9802e532829929971979c2de70e0ec0aa6a76ecf958d10c5016d70477c"} Oct 31 13:10:40.792022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:40.791982 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" event={"ID":"1fe05f4f-3a03-41d7-98e2-abe4c0b13bef","Type":"ContainerStarted","Data":"061482757c8b56dfe5582b50a73cb277fc98b2bad2aca8a5b41dff400b8da199"} Oct 31 13:10:40.792472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:40.792114 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:10:40.813920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:40.813873 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" podStartSLOduration=2.444933435 podStartE2EDuration="5.813860082s" totalImagesPullingTime="3.368926647s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:10:40.812732054 +0000 UTC m=+408.958475874" watchObservedRunningTime="2026-10-31 13:10:40.813860082 +0000 UTC m=+408.959603909" Oct 31 13:10:51.801751 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:10:51.801712 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-7556c48c69-qqdn2" Oct 31 13:11:01.763619 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:11:01.763565 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:11:31.760457 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:11:31.760367 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:12:01.760013 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:12:01.759976 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:12:31.760125 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:12:31.760088 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:13:01.760140 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:13:01.760050 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:13:31.760378 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:13:31.760338 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:14:01.760936 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:14:01.760898 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:14:31.760196 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:14:31.760121 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:15:01.760980 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:15:01.760943 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:15:31.759315 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:15:31.759271 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:16:01.760368 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:16:01.760286 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:16:31.760154 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:16:31.760116 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:17:01.760541 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:17:01.760504 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:17:31.760191 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:17:31.760116 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:18:01.760464 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:18:01.760349 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:18:31.760252 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:18:31.760206 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:19:01.761022 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:19:01.760943 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:19:31.760942 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:19:31.760896 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:20:01.760146 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:20:01.760110 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:20:31.760735 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:20:31.760660 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:20:58.037619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.037566 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:20:58.041387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.041365 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-sbgmp" Oct 31 13:20:58.046111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.046090 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 13:20:58.048315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.048298 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 13:20:58.052877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.052841 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rmxc5\"" Oct 31 13:20:58.054193 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.054166 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:20:58.090551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.090521 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:20:58.146359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.146328 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 13:20:58.162789 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.162760 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4187620e-9bb5-401e-a28e-a0e464c82530-config-file\") pod \"limitador-limitador-7c9b87d574-sbgmp\" (UID: \"4187620e-9bb5-401e-a28e-a0e464c82530\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sbgmp" Oct 31 13:20:58.162946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.162818 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdd5n\" (UniqueName: \"kubernetes.io/projected/4187620e-9bb5-401e-a28e-a0e464c82530-kube-api-access-bdd5n\") pod \"limitador-limitador-7c9b87d574-sbgmp\" (UID: \"4187620e-9bb5-401e-a28e-a0e464c82530\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sbgmp" Oct 31 13:20:58.263481 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.263438 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4187620e-9bb5-401e-a28e-a0e464c82530-config-file\") pod \"limitador-limitador-7c9b87d574-sbgmp\" (UID: \"4187620e-9bb5-401e-a28e-a0e464c82530\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sbgmp" Oct 31 13:20:58.263683 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.263497 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdd5n\" (UniqueName: \"kubernetes.io/projected/4187620e-9bb5-401e-a28e-a0e464c82530-kube-api-access-bdd5n\") pod \"limitador-limitador-7c9b87d574-sbgmp\" (UID: \"4187620e-9bb5-401e-a28e-a0e464c82530\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sbgmp" Oct 31 13:20:58.264197 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.264177 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/4187620e-9bb5-401e-a28e-a0e464c82530-config-file\") pod \"limitador-limitador-7c9b87d574-sbgmp\" (UID: \"4187620e-9bb5-401e-a28e-a0e464c82530\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sbgmp" Oct 31 13:20:58.277133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.277098 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdd5n\" (UniqueName: \"kubernetes.io/projected/4187620e-9bb5-401e-a28e-a0e464c82530-kube-api-access-bdd5n\") pod \"limitador-limitador-7c9b87d574-sbgmp\" (UID: \"4187620e-9bb5-401e-a28e-a0e464c82530\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-sbgmp" Oct 31 13:20:58.353910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.353827 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-sbgmp" Oct 31 13:20:58.631879 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.631798 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:20:58.634931 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:20:58.634897 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4187620e_9bb5_401e_a28e_a0e464c82530.slice/crio-907c28e280837113d5e9aca37c6b95ec09316931e6eba00ab54a7d88f62335b7 WatchSource:0}: Error finding container 907c28e280837113d5e9aca37c6b95ec09316931e6eba00ab54a7d88f62335b7: Status 404 returned error can't find the container with id 907c28e280837113d5e9aca37c6b95ec09316931e6eba00ab54a7d88f62335b7 Oct 31 13:20:58.637040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:58.637019 5532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:20:59.445258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:20:59.445220 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-sbgmp" event={"ID":"4187620e-9bb5-401e-a28e-a0e464c82530","Type":"ContainerStarted","Data":"907c28e280837113d5e9aca37c6b95ec09316931e6eba00ab54a7d88f62335b7"} Oct 31 13:21:01.455758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:21:01.455724 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-sbgmp" event={"ID":"4187620e-9bb5-401e-a28e-a0e464c82530","Type":"ContainerStarted","Data":"83171f833919878bc4c8e77d7e462b3b66026ce68b0dc86c90885623cf70093b"} Oct 31 13:21:01.456130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:21:01.455856 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-sbgmp" Oct 31 13:21:01.475823 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:21:01.475780 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-sbgmp" podStartSLOduration=1.031344118 podStartE2EDuration="3.475766869s" totalImagesPullingTime="2.444422751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:20:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:21:01.475704654 +0000 UTC m=+1029.621448484" watchObservedRunningTime="2026-10-31 13:21:01.475766869 +0000 UTC m=+1029.621510696" Oct 31 13:21:01.760455 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:21:01.760370 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:21:12.466976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:21:12.466941 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-sbgmp" Oct 31 13:21:31.760104 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:21:31.760065 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:22:01.760458 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:22:01.760379 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:22:15.193272 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:15.193237 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:22:31.760008 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:22:31.759971 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:22:53.713752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.713716 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz"] Oct 31 13:22:53.717495 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.717476 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.724922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.724904 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 31 13:22:53.732940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.732916 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz"] Oct 31 13:22:53.773473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.773441 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75407a66-4806-493d-a535-bfaaaddfbe16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.773473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.773478 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.773700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.773528 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75407a66-4806-493d-a535-bfaaaddfbe16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.773700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.773628 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs2n8\" (UniqueName: \"kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-kube-api-access-gs2n8\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.773700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.773672 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.773796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.773768 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.773796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.773793 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.773875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.773821 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.773875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.773838 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.802135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.802111 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-r8f7r\"" Oct 31 13:22:53.874431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.874392 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.874631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.874437 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.874631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.874476 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.874631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.874498 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.874631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.874543 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75407a66-4806-493d-a535-bfaaaddfbe16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.874631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.874573 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.874911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.874670 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75407a66-4806-493d-a535-bfaaaddfbe16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.874911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.874698 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs2n8\" (UniqueName: \"kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-kube-api-access-gs2n8\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.874911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.874751 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.874911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.874871 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.875110 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.875064 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.875110 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.875096 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.875243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.875222 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.875443 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.875425 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75407a66-4806-493d-a535-bfaaaddfbe16-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.876861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.876839 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.876970 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.876951 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75407a66-4806-493d-a535-bfaaaddfbe16-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.888179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.888157 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:53.888390 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:53.888369 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs2n8\" (UniqueName: \"kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-kube-api-access-gs2n8\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-dlwlz\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:54.031244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:54.031167 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:22:54.311966 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:54.311938 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz"] Oct 31 13:22:54.314396 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:22:54.314366 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75407a66_4806_493d_a535_bfaaaddfbe16.slice/crio-5c0a3b361c22aa128191aa3d36b89769d73ad3546ad0028d2d13a6e100aa32d9 WatchSource:0}: Error finding container 5c0a3b361c22aa128191aa3d36b89769d73ad3546ad0028d2d13a6e100aa32d9: Status 404 returned error can't find the container with id 5c0a3b361c22aa128191aa3d36b89769d73ad3546ad0028d2d13a6e100aa32d9 Oct 31 13:22:54.970548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:22:54.970513 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" event={"ID":"75407a66-4806-493d-a535-bfaaaddfbe16","Type":"ContainerStarted","Data":"5c0a3b361c22aa128191aa3d36b89769d73ad3546ad0028d2d13a6e100aa32d9"} Oct 31 13:23:00.178732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:00.178699 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:00.179034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:00.178786 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:00.179034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:00.178814 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:01.000254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:01.000216 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" event={"ID":"75407a66-4806-493d-a535-bfaaaddfbe16","Type":"ContainerStarted","Data":"730bc018d3138d424b7ee1d4dffdf25b9a8c4c03fe21a761932e721e1bc73394"} Oct 31 13:23:01.031436 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:01.031405 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:23:01.032932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:01.032907 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 13:23:01.032932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:01.032904 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" podStartSLOduration=2.171345603 podStartE2EDuration="8.032890863s" totalImagesPullingTime="5.86154526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:01.030338753 +0000 UTC m=+1149.176082580" watchObservedRunningTime="2026-10-31 13:23:01.032890863 +0000 UTC m=+1149.178634691" Oct 31 13:23:01.033123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:01.032949 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" podUID="75407a66-4806-493d-a535-bfaaaddfbe16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 13:23:01.785433 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:23:01.785330 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:23:02.032169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:02.032138 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 13:23:02.032334 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:02.032182 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" podUID="75407a66-4806-493d-a535-bfaaaddfbe16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 13:23:03.032274 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:03.032236 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 13:23:03.032675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:03.032299 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" podUID="75407a66-4806-493d-a535-bfaaaddfbe16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 13:23:04.031930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:04.031899 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 13:23:04.032099 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:04.031955 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" podUID="75407a66-4806-493d-a535-bfaaaddfbe16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 13:23:05.032516 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:05.032479 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 13:23:05.033020 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:05.032540 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" podUID="75407a66-4806-493d-a535-bfaaaddfbe16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 13:23:06.032356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:06.032322 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 31 13:23:06.032534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:06.032373 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" podUID="75407a66-4806-493d-a535-bfaaaddfbe16" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 31 13:23:07.035792 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:07.035762 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:23:07.036268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:07.036132 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:23:07.036704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:07.036684 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:23:08.723318 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.719558 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq"] Oct 31 13:23:08.724476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.724452 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.726549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.726520 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.726695 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.726573 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.726695 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.726652 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.726804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.726747 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.726804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.726784 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.726897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.726826 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.726897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.726855 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.726988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.726902 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.726988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.726930 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8k8f\" (UniqueName: \"kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-kube-api-access-v8k8f\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.735526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.735499 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq"] Oct 31 13:23:08.756481 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.756443 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7"] Oct 31 13:23:08.760232 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.760207 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.775837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.775808 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7"] Oct 31 13:23:08.827853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.827816 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.828036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.827853 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.828036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.827947 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.828036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.827983 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh78v\" (UniqueName: \"kubernetes.io/projected/b13b4423-6b52-4dd7-be75-987fd6e2edfa-kube-api-access-vh78v\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.828036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828004 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.828036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828032 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.828298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828222 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.828298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828242 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.828298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828261 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8k8f\" (UniqueName: \"kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-kube-api-access-v8k8f\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.828464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828418 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.828464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828450 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.828576 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828463 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.828576 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828534 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.828576 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828558 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.828778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828587 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.828778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828616 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.828778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828681 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.828778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828718 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.828966 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828809 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.828966 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828821 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.828966 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828846 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.828966 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828868 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.828966 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.828920 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.830483 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.830460 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.830544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.830468 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.841368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.841339 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8k8f\" (UniqueName: \"kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-kube-api-access-v8k8f\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.841368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.841344 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-vl7kq\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:08.929903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.929862 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.929903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.929903 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.930109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.929931 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.930109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.929966 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.930109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.929993 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.930109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.930035 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh78v\" (UniqueName: \"kubernetes.io/projected/b13b4423-6b52-4dd7-be75-987fd6e2edfa-kube-api-access-vh78v\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.930320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.930167 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.930320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.930249 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.930320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.930304 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.930477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.930415 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.930543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.930496 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.930640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.930564 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.930748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.930727 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.930894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.930874 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.932281 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.932256 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.932521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.932503 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.943908 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.943877 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b13b4423-6b52-4dd7-be75-987fd6e2edfa-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:08.944005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:08.943944 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh78v\" (UniqueName: \"kubernetes.io/projected/b13b4423-6b52-4dd7-be75-987fd6e2edfa-kube-api-access-vh78v\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-hdtr7\" (UID: \"b13b4423-6b52-4dd7-be75-987fd6e2edfa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:09.040157 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:09.040081 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:09.073472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:09.073436 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:09.320801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:09.320774 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq"] Oct 31 13:23:09.323388 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:23:09.323363 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cb4873b_36ee_48b7_aa65_6966c54d12ec.slice/crio-91080f49d1e9bab52a230e2eaef0d2880dd189f82a6f35af5c849be250041e25 WatchSource:0}: Error finding container 91080f49d1e9bab52a230e2eaef0d2880dd189f82a6f35af5c849be250041e25: Status 404 returned error can't find the container with id 91080f49d1e9bab52a230e2eaef0d2880dd189f82a6f35af5c849be250041e25 Oct 31 13:23:09.326402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:09.326367 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:09.326474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:09.326434 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:09.326474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:09.326467 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:09.368363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:09.368299 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7"] Oct 31 13:23:09.370619 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:23:09.370432 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13b4423_6b52_4dd7_be75_987fd6e2edfa.slice/crio-423ea036dbbe16f7e9fd4cd32fe7c9a471b2656768bf8d5cb8689d9669a9e540 WatchSource:0}: Error finding container 423ea036dbbe16f7e9fd4cd32fe7c9a471b2656768bf8d5cb8689d9669a9e540: Status 404 returned error can't find the container with id 423ea036dbbe16f7e9fd4cd32fe7c9a471b2656768bf8d5cb8689d9669a9e540 Oct 31 13:23:09.373492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:09.373464 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:09.373559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:09.373526 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:09.373559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:09.373556 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:10.037673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:10.037639 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" event={"ID":"2cb4873b-36ee-48b7-aa65-6966c54d12ec","Type":"ContainerStarted","Data":"003558e40cf03ea63461e337b426270d71284479b894262eb6533361f4b012fb"} Oct 31 13:23:10.038053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:10.037692 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" event={"ID":"2cb4873b-36ee-48b7-aa65-6966c54d12ec","Type":"ContainerStarted","Data":"91080f49d1e9bab52a230e2eaef0d2880dd189f82a6f35af5c849be250041e25"} Oct 31 13:23:10.048093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:10.041666 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:10.051968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:10.051936 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:23:10.052236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:10.052201 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" podUID="2cb4873b-36ee-48b7-aa65-6966c54d12ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 31 13:23:10.061616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:10.055846 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" event={"ID":"b13b4423-6b52-4dd7-be75-987fd6e2edfa","Type":"ContainerStarted","Data":"785a24f00bf5c7a8f61f8076bf7cb34fc84d27bee778dac03f74a5420d6e4838"} Oct 31 13:23:10.061616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:10.055929 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" event={"ID":"b13b4423-6b52-4dd7-be75-987fd6e2edfa","Type":"ContainerStarted","Data":"423ea036dbbe16f7e9fd4cd32fe7c9a471b2656768bf8d5cb8689d9669a9e540"} Oct 31 13:23:10.084559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:10.084519 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:10.088655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:10.086674 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:23:10.088655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:10.086728 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" podUID="b13b4423-6b52-4dd7-be75-987fd6e2edfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 31 13:23:10.105902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:10.105855 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" podStartSLOduration=2.105838001 podStartE2EDuration="2.105838001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:10.078360755 +0000 UTC m=+1158.224104586" watchObservedRunningTime="2026-10-31 13:23:10.105838001 +0000 UTC m=+1158.251581830" Oct 31 13:23:10.106056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:10.105962 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" podStartSLOduration=2.105957375 podStartE2EDuration="2.105957375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:10.104919321 +0000 UTC m=+1158.250663150" watchObservedRunningTime="2026-10-31 13:23:10.105957375 +0000 UTC m=+1158.251701202" Oct 31 13:23:11.043038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:11.042998 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:23:11.043668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:11.043633 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" podUID="2cb4873b-36ee-48b7-aa65-6966c54d12ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 31 13:23:11.078618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:11.075099 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:23:11.078618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:11.075155 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" podUID="b13b4423-6b52-4dd7-be75-987fd6e2edfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 31 13:23:12.054620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:12.052009 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:23:12.054620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:12.052074 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" podUID="2cb4873b-36ee-48b7-aa65-6966c54d12ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 31 13:23:12.076616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:12.076154 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:23:12.076616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:12.076213 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" podUID="b13b4423-6b52-4dd7-be75-987fd6e2edfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 31 13:23:13.043048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:13.043004 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:23:13.043326 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:13.043294 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" podUID="2cb4873b-36ee-48b7-aa65-6966c54d12ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 31 13:23:13.076607 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:13.076547 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:23:13.077189 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:13.077128 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" podUID="b13b4423-6b52-4dd7-be75-987fd6e2edfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 31 13:23:14.054619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:14.051125 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:23:14.054619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:14.051203 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" podUID="2cb4873b-36ee-48b7-aa65-6966c54d12ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 31 13:23:14.075374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:14.074962 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:23:14.075374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:14.075021 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" podUID="b13b4423-6b52-4dd7-be75-987fd6e2edfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 31 13:23:15.043766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:15.043728 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Oct 31 13:23:15.044337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:15.044305 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" podUID="2cb4873b-36ee-48b7-aa65-6966c54d12ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Oct 31 13:23:15.076119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:15.076086 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Oct 31 13:23:15.076393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:15.076355 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" podUID="b13b4423-6b52-4dd7-be75-987fd6e2edfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Oct 31 13:23:16.044876 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:16.044850 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:16.045248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:16.045103 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:16.045724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:16.045706 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:23:16.077553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:16.077530 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:16.077822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:16.077803 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:16.078380 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:16.078366 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" Oct 31 13:23:17.833195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:17.833162 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:23:19.495062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:19.495031 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:23:23.767206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:23.767167 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw"] Oct 31 13:23:23.779426 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:23.779392 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:23.783005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:23.782975 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw"] Oct 31 13:23:23.930271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:23.930234 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj4ht\" (UniqueName: \"kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-kube-api-access-vj4ht\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:23.930271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:23.930273 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:23.930492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:23.930295 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:23.930492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:23.930463 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:23.930557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:23.930500 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:23.930557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:23.930519 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:23.930646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:23.930586 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:23.930703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:23.930690 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:23.930737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:23.930718 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.031575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.031494 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.031575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.031540 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.031575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.031576 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.031889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.031615 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.031889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.031662 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.031889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.031685 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.031889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.031745 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj4ht\" (UniqueName: \"kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-kube-api-access-vj4ht\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.031889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.031867 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.032138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.031892 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.032138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.032123 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.032247 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.032197 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.032307 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.032260 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.032441 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.032420 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.032615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.032567 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.033936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.033919 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.034322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.034305 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.045583 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.045558 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.045919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.045897 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj4ht\" (UniqueName: \"kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-kube-api-access-vj4ht\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-srpzw\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.095514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.095479 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:24.377204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.377181 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw"] Oct 31 13:23:24.379649 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:23:24.379618 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fb7e7a8_5247_4e9d_966a_7a4be7c06673.slice/crio-339de1adf565ff16192d4821805408550a5f9c0f0d1c4897a7a6f013fd4503f9 WatchSource:0}: Error finding container 339de1adf565ff16192d4821805408550a5f9c0f0d1c4897a7a6f013fd4503f9: Status 404 returned error can't find the container with id 339de1adf565ff16192d4821805408550a5f9c0f0d1c4897a7a6f013fd4503f9 Oct 31 13:23:24.381932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.381903 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:24.382005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.381965 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:24.382005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.381991 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:23:24.396183 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.396156 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft"] Oct 31 13:23:24.401711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.401693 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.410746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.410580 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 31 13:23:24.410746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.410717 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-9845x\"" Oct 31 13:23:24.411032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.411010 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft"] Oct 31 13:23:24.430359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.430334 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 13:23:24.430541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.430469 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 13:23:24.532658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.532628 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:23:24.537651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.537617 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b01eeee-1f42-4e8e-af23-4e228345c60f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.537841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.537726 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b01eeee-1f42-4e8e-af23-4e228345c60f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.540104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.537855 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b01eeee-1f42-4e8e-af23-4e228345c60f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.540104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.537889 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b01eeee-1f42-4e8e-af23-4e228345c60f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.540104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.538011 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrh2q\" (UniqueName: \"kubernetes.io/projected/0b01eeee-1f42-4e8e-af23-4e228345c60f-kube-api-access-rrh2q\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.540104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.538058 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b01eeee-1f42-4e8e-af23-4e228345c60f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.639095 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.639019 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b01eeee-1f42-4e8e-af23-4e228345c60f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.639095 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.639053 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b01eeee-1f42-4e8e-af23-4e228345c60f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.639340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.639105 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b01eeee-1f42-4e8e-af23-4e228345c60f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.639340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.639235 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b01eeee-1f42-4e8e-af23-4e228345c60f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.639561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.639346 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrh2q\" (UniqueName: \"kubernetes.io/projected/0b01eeee-1f42-4e8e-af23-4e228345c60f-kube-api-access-rrh2q\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.639561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.639394 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b01eeee-1f42-4e8e-af23-4e228345c60f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.639561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.639479 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b01eeee-1f42-4e8e-af23-4e228345c60f-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.639760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.639704 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b01eeee-1f42-4e8e-af23-4e228345c60f-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.639819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.639758 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b01eeee-1f42-4e8e-af23-4e228345c60f-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.641754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.641726 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b01eeee-1f42-4e8e-af23-4e228345c60f-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.642022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.642004 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b01eeee-1f42-4e8e-af23-4e228345c60f-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.653227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.653204 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrh2q\" (UniqueName: \"kubernetes.io/projected/0b01eeee-1f42-4e8e-af23-4e228345c60f-kube-api-access-rrh2q\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-sngft\" (UID: \"0b01eeee-1f42-4e8e-af23-4e228345c60f\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:24.732748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:24.732718 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:25.026073 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:25.026038 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft"] Oct 31 13:23:25.030548 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:23:25.030516 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b01eeee_1f42_4e8e_af23_4e228345c60f.slice/crio-77750ed76a1337f929d16d73c84dcebd530466f525af0585952a2e0cc8472883 WatchSource:0}: Error finding container 77750ed76a1337f929d16d73c84dcebd530466f525af0585952a2e0cc8472883: Status 404 returned error can't find the container with id 77750ed76a1337f929d16d73c84dcebd530466f525af0585952a2e0cc8472883 Oct 31 13:23:25.176760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:25.176712 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" event={"ID":"3fb7e7a8-5247-4e9d-966a-7a4be7c06673","Type":"ContainerStarted","Data":"b31d278fad14c28399cb16b358c33b520b368ac30e09465ffa19384b7f4e1838"} Oct 31 13:23:25.176976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:25.176790 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" event={"ID":"3fb7e7a8-5247-4e9d-966a-7a4be7c06673","Type":"ContainerStarted","Data":"339de1adf565ff16192d4821805408550a5f9c0f0d1c4897a7a6f013fd4503f9"} Oct 31 13:23:25.179837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:25.179805 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" event={"ID":"0b01eeee-1f42-4e8e-af23-4e228345c60f","Type":"ContainerStarted","Data":"77750ed76a1337f929d16d73c84dcebd530466f525af0585952a2e0cc8472883"} Oct 31 13:23:25.205546 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:25.205484 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podStartSLOduration=2.205466063 podStartE2EDuration="2.205466063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:25.203498756 +0000 UTC m=+1173.349242585" watchObservedRunningTime="2026-10-31 13:23:25.205466063 +0000 UTC m=+1173.351209892" Oct 31 13:23:26.103617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.103066 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:26.103617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.103177 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 13:23:26.103617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.103218 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 13:23:26.384876 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.384772 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264"] Oct 31 13:23:26.391760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.391732 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.400028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.400000 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 31 13:23:26.401486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.401462 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264"] Oct 31 13:23:26.524815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.524779 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5001cfd-6512-441f-a9d9-0b566d4f06b2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.524994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.524849 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5001cfd-6512-441f-a9d9-0b566d4f06b2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.524994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.524894 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5001cfd-6512-441f-a9d9-0b566d4f06b2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.524994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.524928 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmxb6\" (UniqueName: \"kubernetes.io/projected/e5001cfd-6512-441f-a9d9-0b566d4f06b2-kube-api-access-fmxb6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.524994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.524968 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5001cfd-6512-441f-a9d9-0b566d4f06b2-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.525212 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.525061 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5001cfd-6512-441f-a9d9-0b566d4f06b2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.626071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.626034 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5001cfd-6512-441f-a9d9-0b566d4f06b2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.626266 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.626083 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5001cfd-6512-441f-a9d9-0b566d4f06b2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.626266 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.626130 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5001cfd-6512-441f-a9d9-0b566d4f06b2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.626266 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.626178 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5001cfd-6512-441f-a9d9-0b566d4f06b2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.626266 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.626222 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmxb6\" (UniqueName: \"kubernetes.io/projected/e5001cfd-6512-441f-a9d9-0b566d4f06b2-kube-api-access-fmxb6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.626497 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.626267 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5001cfd-6512-441f-a9d9-0b566d4f06b2-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.626692 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.626668 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5001cfd-6512-441f-a9d9-0b566d4f06b2-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.626822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.626720 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5001cfd-6512-441f-a9d9-0b566d4f06b2-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.626822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.626722 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5001cfd-6512-441f-a9d9-0b566d4f06b2-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.629223 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.629197 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5001cfd-6512-441f-a9d9-0b566d4f06b2-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.631022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.630946 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5001cfd-6512-441f-a9d9-0b566d4f06b2-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.685324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.685278 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmxb6\" (UniqueName: \"kubernetes.io/projected/e5001cfd-6512-441f-a9d9-0b566d4f06b2-kube-api-access-fmxb6\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264\" (UID: \"e5001cfd-6512-441f-a9d9-0b566d4f06b2\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:26.708983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:26.708945 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:27.051673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:27.050231 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264"] Oct 31 13:23:27.096091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:27.096063 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 13:23:27.096229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:27.096108 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 13:23:27.192697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:27.192652 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" event={"ID":"e5001cfd-6512-441f-a9d9-0b566d4f06b2","Type":"ContainerStarted","Data":"c80a1a0b02d1d4143a42f4e518ad3b60b558e210bdb9e75265d061da7ee22e74"} Oct 31 13:23:28.098619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:28.096392 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 13:23:28.098619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:28.096477 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 13:23:28.361942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:28.360222 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:23:29.096240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:29.096212 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 13:23:29.096340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:29.096272 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 13:23:29.221258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:29.221219 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" event={"ID":"0b01eeee-1f42-4e8e-af23-4e228345c60f","Type":"ContainerStarted","Data":"bd3b637a653740be02ae95737031eaeb98a8cba117efe4766969196ab76b5e27"} Oct 31 13:23:29.223612 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:29.223567 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" event={"ID":"e5001cfd-6512-441f-a9d9-0b566d4f06b2","Type":"ContainerStarted","Data":"7ab00d24f03f72ef9759dede6a8fa8c98690a1e752c6d7ba43b8acb56de2409a"} Oct 31 13:23:29.273705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:29.272275 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" podStartSLOduration=1.817069539 podStartE2EDuration="5.272254744s" totalImagesPullingTime="3.455185205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:29.247483616 +0000 UTC m=+1177.393227447" watchObservedRunningTime="2026-10-31 13:23:29.272254744 +0000 UTC m=+1177.417998572" Oct 31 13:23:29.712620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:29.711676 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:29.732922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:29.732882 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:30.096140 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:30.096018 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" start-of-body= Oct 31 13:23:30.096528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:30.096488 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": dial tcp 10.132.0.37:15021: connect: connection refused" Oct 31 13:23:31.888436 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:23:31.888363 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:23:32.142679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:32.139700 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:23:32.142679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:32.139767 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:23:33.140615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.140543 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:23:33.141016 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.140630 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:23:33.180960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.180909 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podStartSLOduration=5.275322909 podStartE2EDuration="7.18089444s" totalImagesPullingTime="1.905571531s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:29.274020827 +0000 UTC m=+1177.419764657" watchObservedRunningTime="2026-10-31 13:23:33.18089444 +0000 UTC m=+1181.326638268" Oct 31 13:23:33.181494 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.181474 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:23:33.735474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.734058 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w"] Oct 31 13:23:33.738040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.738015 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.761616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.756923 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 31 13:23:33.761616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.760941 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w"] Oct 31 13:23:33.865168 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.865129 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f952465-49ad-43cf-9598-dd3741e8de2c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.865512 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.865484 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f952465-49ad-43cf-9598-dd3741e8de2c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.865788 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.865765 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq8ng\" (UniqueName: \"kubernetes.io/projected/3f952465-49ad-43cf-9598-dd3741e8de2c-kube-api-access-lq8ng\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.865940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.865924 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f952465-49ad-43cf-9598-dd3741e8de2c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.866091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.866073 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f952465-49ad-43cf-9598-dd3741e8de2c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.866265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.866248 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f952465-49ad-43cf-9598-dd3741e8de2c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.966873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.966837 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f952465-49ad-43cf-9598-dd3741e8de2c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.967172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.967141 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f952465-49ad-43cf-9598-dd3741e8de2c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.967299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.967187 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f952465-49ad-43cf-9598-dd3741e8de2c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.967299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.967247 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f952465-49ad-43cf-9598-dd3741e8de2c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.967299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.967253 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f952465-49ad-43cf-9598-dd3741e8de2c-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.967485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.967324 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq8ng\" (UniqueName: \"kubernetes.io/projected/3f952465-49ad-43cf-9598-dd3741e8de2c-kube-api-access-lq8ng\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.967485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.967376 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f952465-49ad-43cf-9598-dd3741e8de2c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.967617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.967572 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f952465-49ad-43cf-9598-dd3741e8de2c-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.967870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.967846 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f952465-49ad-43cf-9598-dd3741e8de2c-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.970532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.970505 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f952465-49ad-43cf-9598-dd3741e8de2c-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.971055 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.971032 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f952465-49ad-43cf-9598-dd3741e8de2c-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:33.981534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:33.981499 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq8ng\" (UniqueName: \"kubernetes.io/projected/3f952465-49ad-43cf-9598-dd3741e8de2c-kube-api-access-lq8ng\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-h4g6w\" (UID: \"3f952465-49ad-43cf-9598-dd3741e8de2c\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:34.063229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:34.063127 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:34.142663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:34.141675 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:23:34.142663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:34.141752 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:23:34.375472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:34.375428 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w"] Oct 31 13:23:34.379141 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:23:34.379107 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f952465_49ad_43cf_9598_dd3741e8de2c.slice/crio-6d59115957785d68b1364ff0cf60f260e4d16c33871a0b61f6b18307515572af WatchSource:0}: Error finding container 6d59115957785d68b1364ff0cf60f260e4d16c33871a0b61f6b18307515572af: Status 404 returned error can't find the container with id 6d59115957785d68b1364ff0cf60f260e4d16c33871a0b61f6b18307515572af Oct 31 13:23:34.742148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:34.742032 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:34.835025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:34.834986 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:35.193617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:35.189314 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:23:35.193617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:35.189390 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:23:35.324100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:35.323981 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" event={"ID":"3f952465-49ad-43cf-9598-dd3741e8de2c","Type":"ContainerStarted","Data":"3f3bcdc3d86249de14a3574e0ac59ab10f7446a99d82b7cac449f5ba3acb9c34"} Oct 31 13:23:35.324346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:35.324322 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" event={"ID":"3f952465-49ad-43cf-9598-dd3741e8de2c","Type":"ContainerStarted","Data":"6d59115957785d68b1364ff0cf60f260e4d16c33871a0b61f6b18307515572af"} Oct 31 13:23:35.379392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:35.379344 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" Oct 31 13:23:35.432327 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:35.432262 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" podStartSLOduration=1.9205644149999999 podStartE2EDuration="2.432238791s" totalImagesPullingTime="511.674376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:23:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:23:35.368402835 +0000 UTC m=+1183.514146664" watchObservedRunningTime="2026-10-31 13:23:35.432238791 +0000 UTC m=+1183.577982623" Oct 31 13:23:36.064619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:36.064560 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:36.190517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:36.190363 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:23:36.190517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:36.190440 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:23:36.710243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:36.709462 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:36.730588 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:36.730555 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:37.235368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:37.232701 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:23:37.235368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:37.232766 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:23:37.388123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:37.388089 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:23:37.429443 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:37.429403 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:23:38.233771 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:38.233732 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:23:38.234343 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:38.233797 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:23:39.095987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:39.095952 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:39.276822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:39.276772 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:23:39.277325 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:39.276829 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:23:40.277720 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:40.277671 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:23:40.278099 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:40.277752 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:23:41.318752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:41.318712 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:23:41.319151 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:41.318794 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:23:42.075896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:42.075862 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:42.077221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:42.077197 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:23:44.063406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:44.063364 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:44.075662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:44.075637 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:23:44.382214 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:23:44.382127 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" Oct 31 13:24:01.760526 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:24:01.760484 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:24:31.760879 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:24:31.760840 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:24:43.012747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.012716 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj"] Oct 31 13:24:43.017763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.017733 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.026647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.026625 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-7b3ddd-openshift-default-dockercfg-lzzgx\"" Oct 31 13:24:43.034658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.034629 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj"] Oct 31 13:24:43.118231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.118187 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.118474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.118313 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.118474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.118364 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.118474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.118397 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.118660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.118530 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928c06ae-3fb5-4d89-aa1f-0da90832da40-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.118738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.118714 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6kt\" (UniqueName: \"kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-kube-api-access-rc6kt\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.118885 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.118861 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.118949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.118909 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.119055 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.119032 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.220624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.220569 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928c06ae-3fb5-4d89-aa1f-0da90832da40-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.220869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.220841 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6kt\" (UniqueName: \"kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-kube-api-access-rc6kt\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.221000 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.220925 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.221000 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.220952 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.221000 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.220970 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.221164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.221024 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.221164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.221064 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.221164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.221100 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.221164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.221125 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.221353 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.221333 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928c06ae-3fb5-4d89-aa1f-0da90832da40-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.221405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.221380 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.221747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.221716 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.221852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.221799 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.221852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.221800 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.223745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.223722 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.224284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.224258 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.233479 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.233457 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476"] Oct 31 13:24:43.237360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.237344 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.246040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.246017 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.252690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.252666 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6kt\" (UniqueName: \"kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-kube-api-access-rc6kt\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.254713 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.254695 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-8b2f0a-openshift-default-dockercfg-w6shk\"" Oct 31 13:24:43.271172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.271148 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476"] Oct 31 13:24:43.322135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.322101 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkhnw\" (UniqueName: \"kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-kube-api-access-wkhnw\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.322310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.322181 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.322310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.322264 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.322310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.322294 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.322509 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.322488 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.322572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.322542 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.322721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.322663 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.322721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.322696 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.322805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.322768 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.336866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.336630 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:43.424251 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.424210 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkhnw\" (UniqueName: \"kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-kube-api-access-wkhnw\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.424417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.424296 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.424417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.424331 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.424417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.424363 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.424577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.424451 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.424577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.424500 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.424577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.424543 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.424724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.424573 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.424724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.424636 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.425150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.425119 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.425747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.425725 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.426007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.425985 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.426292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.426271 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.427186 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.427168 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.429299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.429266 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.433304 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.433278 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.444920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.444883 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkhnw\" (UniqueName: \"kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-kube-api-access-wkhnw\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.445017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.444995 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.509703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.509671 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq"] Oct 31 13:24:43.515405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.515382 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.526736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.526710 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq"] Oct 31 13:24:43.562912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.562878 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-d85d82-openshift-default-dockercfg-xn9td\"" Oct 31 13:24:43.577045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.577019 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:43.626579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.626554 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj"] Oct 31 13:24:43.628731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.628698 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.628842 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.628736 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.628842 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.628764 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.628842 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.628792 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.628842 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.628808 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.629058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.628854 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.629058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.628877 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.629058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.628923 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bdbv\" (UniqueName: \"kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-kube-api-access-9bdbv\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.629058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.628945 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.631887 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.631854 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:43.631970 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.631939 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:43.632065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.631985 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:43.654663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.654630 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" event={"ID":"928c06ae-3fb5-4d89-aa1f-0da90832da40","Type":"ContainerStarted","Data":"4eaf4664008e3c63eb99393b7cb6e4cbc61ceb4ce0187c2f3240cdc075643e8e"} Oct 31 13:24:43.732642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.730005 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.732642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.730663 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.732642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.730576 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.732642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.731728 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.732642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.731806 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.732642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.731833 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.732642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.732542 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.733161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.732772 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.733161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.732809 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.733161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.732887 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bdbv\" (UniqueName: \"kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-kube-api-access-9bdbv\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.733161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.732925 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.733721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.733690 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.733977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.733949 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.734241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.734200 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.735475 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.735430 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.742344 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.742287 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.747401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.747348 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.756049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.756020 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bdbv\" (UniqueName: \"kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-kube-api-access-9bdbv\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.832574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.832500 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:43.859949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.859920 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476"] Oct 31 13:24:43.863801 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:24:43.863757 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7ef42d_5b18_414a_a0ff_492cc511c0db.slice/crio-d7417a7a68a3097f9eebb0b408a9b84dd216ce90895cccf413b434e8230f9865 WatchSource:0}: Error finding container d7417a7a68a3097f9eebb0b408a9b84dd216ce90895cccf413b434e8230f9865: Status 404 returned error can't find the container with id d7417a7a68a3097f9eebb0b408a9b84dd216ce90895cccf413b434e8230f9865 Oct 31 13:24:43.866169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.866128 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:43.866278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.866225 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:43.866278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:43.866258 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:44.130718 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:24:44.130687 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ff1a789_0fbf_43ee_91dc_41eb7636c4e4.slice/crio-cdd304301ae41d1c1444cc2dc871d8a0856ded81b217fc338aa16f7539afb767 WatchSource:0}: Error finding container cdd304301ae41d1c1444cc2dc871d8a0856ded81b217fc338aa16f7539afb767: Status 404 returned error can't find the container with id cdd304301ae41d1c1444cc2dc871d8a0856ded81b217fc338aa16f7539afb767 Oct 31 13:24:44.131107 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:44.131013 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq"] Oct 31 13:24:44.132527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:44.132492 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:44.132674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:44.132571 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:44.132674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:44.132636 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:24:44.660818 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:44.660782 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" event={"ID":"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4","Type":"ContainerStarted","Data":"2735d3ea10d10be65c87f9c9ec36ca13c4c607e7d15a4bde348cdc030d896f9b"} Oct 31 13:24:44.661019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:44.660842 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" event={"ID":"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4","Type":"ContainerStarted","Data":"cdd304301ae41d1c1444cc2dc871d8a0856ded81b217fc338aa16f7539afb767"} Oct 31 13:24:44.662349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:44.662319 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" event={"ID":"928c06ae-3fb5-4d89-aa1f-0da90832da40","Type":"ContainerStarted","Data":"a6db087353b60087ed320ed226d37351b82d2bda2c61292278f84a9a6102f5b3"} Oct 31 13:24:44.663824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:44.663801 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" event={"ID":"8a7ef42d-5b18-414a-a0ff-492cc511c0db","Type":"ContainerStarted","Data":"dd02214c26380e2ef79dacdbf9badce4d57811fff29ca0ce19a075348c5146b4"} Oct 31 13:24:44.663939 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:44.663837 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" event={"ID":"8a7ef42d-5b18-414a-a0ff-492cc511c0db","Type":"ContainerStarted","Data":"d7417a7a68a3097f9eebb0b408a9b84dd216ce90895cccf413b434e8230f9865"} Oct 31 13:24:44.685886 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:44.685833 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" podStartSLOduration=1.685819932 podStartE2EDuration="1.685819932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:44.685532077 +0000 UTC m=+1252.831275907" watchObservedRunningTime="2026-10-31 13:24:44.685819932 +0000 UTC m=+1252.831563757" Oct 31 13:24:44.719255 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:44.719210 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" podStartSLOduration=2.719196378 podStartE2EDuration="2.719196378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:44.717475315 +0000 UTC m=+1252.863219143" watchObservedRunningTime="2026-10-31 13:24:44.719196378 +0000 UTC m=+1252.864940202" Oct 31 13:24:44.744083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:44.744043 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" podStartSLOduration=1.744031686 podStartE2EDuration="1.744031686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:44.742137326 +0000 UTC m=+1252.887881178" watchObservedRunningTime="2026-10-31 13:24:44.744031686 +0000 UTC m=+1252.889775513" Oct 31 13:24:44.833354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:44.833327 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:45.337874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:45.337840 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:45.343159 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:45.343131 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:45.578166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:45.578131 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:45.583549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:45.583520 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:45.669109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:45.669078 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:45.669279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:45.669128 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:45.669813 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:45.669791 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:24:45.669912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:45.669847 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:24:45.838916 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:45.838885 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:46.673609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:46.673575 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:46.674516 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:46.674498 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:24:47.605692 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:47.605657 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q"] Oct 31 13:24:47.610067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:47.610036 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" Oct 31 13:24:47.621141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:47.621116 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-6784d6-dockercfg-t6zsz\"" Oct 31 13:24:47.621285 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:47.621185 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-6784d6\"" Oct 31 13:24:47.623735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:47.623716 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q"] Oct 31 13:24:47.780362 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:47.780326 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndklm\" (UniqueName: \"kubernetes.io/projected/27513f81-9d1f-407b-bca8-86c3d9659bf0-kube-api-access-ndklm\") pod \"payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q\" (UID: \"27513f81-9d1f-407b-bca8-86c3d9659bf0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" Oct 31 13:24:47.780847 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:47.780565 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27513f81-9d1f-407b-bca8-86c3d9659bf0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q\" (UID: \"27513f81-9d1f-407b-bca8-86c3d9659bf0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" Oct 31 13:24:47.881477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:47.881383 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndklm\" (UniqueName: \"kubernetes.io/projected/27513f81-9d1f-407b-bca8-86c3d9659bf0-kube-api-access-ndklm\") pod \"payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q\" (UID: \"27513f81-9d1f-407b-bca8-86c3d9659bf0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" Oct 31 13:24:47.881477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:47.881451 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27513f81-9d1f-407b-bca8-86c3d9659bf0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q\" (UID: \"27513f81-9d1f-407b-bca8-86c3d9659bf0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" Oct 31 13:24:47.882225 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:47.882188 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27513f81-9d1f-407b-bca8-86c3d9659bf0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q\" (UID: \"27513f81-9d1f-407b-bca8-86c3d9659bf0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" Oct 31 13:24:47.894794 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:47.894769 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndklm\" (UniqueName: \"kubernetes.io/projected/27513f81-9d1f-407b-bca8-86c3d9659bf0-kube-api-access-ndklm\") pod \"payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q\" (UID: \"27513f81-9d1f-407b-bca8-86c3d9659bf0\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" Oct 31 13:24:47.925527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:47.925497 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" Oct 31 13:24:48.216826 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:48.216792 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q"] Oct 31 13:24:48.221103 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:24:48.221068 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27513f81_9d1f_407b_bca8_86c3d9659bf0.slice/crio-ee5fb88a0e620b9d6ee05b61ca4d17c0d0aa2f09026240429374738ecda8b11a WatchSource:0}: Error finding container ee5fb88a0e620b9d6ee05b61ca4d17c0d0aa2f09026240429374738ecda8b11a: Status 404 returned error can't find the container with id ee5fb88a0e620b9d6ee05b61ca4d17c0d0aa2f09026240429374738ecda8b11a Oct 31 13:24:48.686550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:48.686512 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" event={"ID":"27513f81-9d1f-407b-bca8-86c3d9659bf0","Type":"ContainerStarted","Data":"ee5fb88a0e620b9d6ee05b61ca4d17c0d0aa2f09026240429374738ecda8b11a"} Oct 31 13:24:48.834723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:48.834681 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs"] Oct 31 13:24:48.839229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:48.839203 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" Oct 31 13:24:48.850553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:48.850522 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs"] Oct 31 13:24:48.892621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:48.892567 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-d85d82\"" Oct 31 13:24:48.892798 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:48.892712 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-d85d82-dockercfg-vwfrq\"" Oct 31 13:24:48.995242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:48.995165 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4765q\" (UniqueName: \"kubernetes.io/projected/6cd8e2bd-93c5-4108-a69a-1648d856fede-kube-api-access-4765q\") pod \"payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs\" (UID: \"6cd8e2bd-93c5-4108-a69a-1648d856fede\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" Oct 31 13:24:48.995434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:48.995408 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cd8e2bd-93c5-4108-a69a-1648d856fede-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs\" (UID: \"6cd8e2bd-93c5-4108-a69a-1648d856fede\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" Oct 31 13:24:49.096810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.096767 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cd8e2bd-93c5-4108-a69a-1648d856fede-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs\" (UID: \"6cd8e2bd-93c5-4108-a69a-1648d856fede\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" Oct 31 13:24:49.096972 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.096831 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4765q\" (UniqueName: \"kubernetes.io/projected/6cd8e2bd-93c5-4108-a69a-1648d856fede-kube-api-access-4765q\") pod \"payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs\" (UID: \"6cd8e2bd-93c5-4108-a69a-1648d856fede\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" Oct 31 13:24:49.097827 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.097799 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cd8e2bd-93c5-4108-a69a-1648d856fede-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs\" (UID: \"6cd8e2bd-93c5-4108-a69a-1648d856fede\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" Oct 31 13:24:49.121496 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.121464 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4765q\" (UniqueName: \"kubernetes.io/projected/6cd8e2bd-93c5-4108-a69a-1648d856fede-kube-api-access-4765q\") pod \"payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs\" (UID: \"6cd8e2bd-93c5-4108-a69a-1648d856fede\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" Oct 31 13:24:49.170421 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.170388 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" Oct 31 13:24:49.224456 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.223575 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv"] Oct 31 13:24:49.229224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.229196 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" Oct 31 13:24:49.245207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.245183 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-7b3ddd\"" Oct 31 13:24:49.246472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.246191 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-7b3ddd-dockercfg-qzwgw\"" Oct 31 13:24:49.248878 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.248831 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv"] Oct 31 13:24:49.299252 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.299220 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq"] Oct 31 13:24:49.302934 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.302913 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" Oct 31 13:24:49.315693 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.315668 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq"] Oct 31 13:24:49.400828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.400795 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93db7894-3327-4e51-8c0a-81a77124dd7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv\" (UID: \"93db7894-3327-4e51-8c0a-81a77124dd7a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" Oct 31 13:24:49.401240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.401199 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np7rq\" (UniqueName: \"kubernetes.io/projected/93db7894-3327-4e51-8c0a-81a77124dd7a-kube-api-access-np7rq\") pod \"payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv\" (UID: \"93db7894-3327-4e51-8c0a-81a77124dd7a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" Oct 31 13:24:49.401706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.401680 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq\" (UID: \"dcb7b65f-cd03-4643-9a7a-e246ddeafe52\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" Oct 31 13:24:49.401888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.401866 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs7kn\" (UniqueName: \"kubernetes.io/projected/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-kube-api-access-fs7kn\") pod \"payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq\" (UID: \"dcb7b65f-cd03-4643-9a7a-e246ddeafe52\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" Oct 31 13:24:49.481550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.481330 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs"] Oct 31 13:24:49.504263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.504163 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93db7894-3327-4e51-8c0a-81a77124dd7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv\" (UID: \"93db7894-3327-4e51-8c0a-81a77124dd7a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" Oct 31 13:24:49.504263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.504260 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np7rq\" (UniqueName: \"kubernetes.io/projected/93db7894-3327-4e51-8c0a-81a77124dd7a-kube-api-access-np7rq\") pod \"payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv\" (UID: \"93db7894-3327-4e51-8c0a-81a77124dd7a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" Oct 31 13:24:49.504495 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.504341 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq\" (UID: \"dcb7b65f-cd03-4643-9a7a-e246ddeafe52\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" Oct 31 13:24:49.504495 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.504367 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs7kn\" (UniqueName: \"kubernetes.io/projected/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-kube-api-access-fs7kn\") pod \"payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq\" (UID: \"dcb7b65f-cd03-4643-9a7a-e246ddeafe52\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" Oct 31 13:24:49.505553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.505526 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93db7894-3327-4e51-8c0a-81a77124dd7a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv\" (UID: \"93db7894-3327-4e51-8c0a-81a77124dd7a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" Oct 31 13:24:49.506457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.506425 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq\" (UID: \"dcb7b65f-cd03-4643-9a7a-e246ddeafe52\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" Oct 31 13:24:49.520661 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.520636 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np7rq\" (UniqueName: \"kubernetes.io/projected/93db7894-3327-4e51-8c0a-81a77124dd7a-kube-api-access-np7rq\") pod \"payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv\" (UID: \"93db7894-3327-4e51-8c0a-81a77124dd7a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" Oct 31 13:24:49.525071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.525043 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs7kn\" (UniqueName: \"kubernetes.io/projected/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-kube-api-access-fs7kn\") pod \"payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq\" (UID: \"dcb7b65f-cd03-4643-9a7a-e246ddeafe52\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" Oct 31 13:24:49.558249 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.558206 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" Oct 31 13:24:49.618292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.618197 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" Oct 31 13:24:49.711870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.711767 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" event={"ID":"6cd8e2bd-93c5-4108-a69a-1648d856fede","Type":"ContainerStarted","Data":"c82c0ee168f498c1b4d8cde17b8785cb8fe14faefda507f4deedee09b8605668"} Oct 31 13:24:49.803035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.802950 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg"] Oct 31 13:24:49.810022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.809992 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" Oct 31 13:24:49.822876 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.822832 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-8b2f0a-dockercfg-qh48q\"" Oct 31 13:24:49.825907 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.825837 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg"] Oct 31 13:24:49.861417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.861391 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv"] Oct 31 13:24:49.863925 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:24:49.863895 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93db7894_3327_4e51_8c0a_81a77124dd7a.slice/crio-01307333f4cee48d59ecdb2d5c91e8709282036597a2a19303950301633038f2 WatchSource:0}: Error finding container 01307333f4cee48d59ecdb2d5c91e8709282036597a2a19303950301633038f2: Status 404 returned error can't find the container with id 01307333f4cee48d59ecdb2d5c91e8709282036597a2a19303950301633038f2 Oct 31 13:24:49.881798 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.881771 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-8b2f0a\"" Oct 31 13:24:49.908490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.908439 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e44283dd-c621-4479-bbd2-b392cfe2a176-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg\" (UID: \"e44283dd-c621-4479-bbd2-b392cfe2a176\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" Oct 31 13:24:49.908688 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.908620 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92s5n\" (UniqueName: \"kubernetes.io/projected/e44283dd-c621-4479-bbd2-b392cfe2a176-kube-api-access-92s5n\") pod \"payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg\" (UID: \"e44283dd-c621-4479-bbd2-b392cfe2a176\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" Oct 31 13:24:49.920620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:49.920569 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq"] Oct 31 13:24:49.922583 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:24:49.922552 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb7b65f_cd03_4643_9a7a_e246ddeafe52.slice/crio-148f0c190da84b46f408ef4a362bd4644ab79b82e1c68321ca01dc0abdbb12a2 WatchSource:0}: Error finding container 148f0c190da84b46f408ef4a362bd4644ab79b82e1c68321ca01dc0abdbb12a2: Status 404 returned error can't find the container with id 148f0c190da84b46f408ef4a362bd4644ab79b82e1c68321ca01dc0abdbb12a2 Oct 31 13:24:50.013616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.012529 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92s5n\" (UniqueName: \"kubernetes.io/projected/e44283dd-c621-4479-bbd2-b392cfe2a176-kube-api-access-92s5n\") pod \"payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg\" (UID: \"e44283dd-c621-4479-bbd2-b392cfe2a176\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" Oct 31 13:24:50.013616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.012778 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e44283dd-c621-4479-bbd2-b392cfe2a176-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg\" (UID: \"e44283dd-c621-4479-bbd2-b392cfe2a176\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" Oct 31 13:24:50.014094 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.014068 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e44283dd-c621-4479-bbd2-b392cfe2a176-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg\" (UID: \"e44283dd-c621-4479-bbd2-b392cfe2a176\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" Oct 31 13:24:50.052396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.052354 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92s5n\" (UniqueName: \"kubernetes.io/projected/e44283dd-c621-4479-bbd2-b392cfe2a176-kube-api-access-92s5n\") pod \"payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg\" (UID: \"e44283dd-c621-4479-bbd2-b392cfe2a176\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" Oct 31 13:24:50.129183 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.128932 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" Oct 31 13:24:50.451566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.451473 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg"] Oct 31 13:24:50.456937 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:24:50.456895 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode44283dd_c621_4479_bbd2_b392cfe2a176.slice/crio-7a16c0ca2dfe92930cb3c4b29d82a5bba6e2026969ecfa9bcd8b8682fa60deb0 WatchSource:0}: Error finding container 7a16c0ca2dfe92930cb3c4b29d82a5bba6e2026969ecfa9bcd8b8682fa60deb0: Status 404 returned error can't find the container with id 7a16c0ca2dfe92930cb3c4b29d82a5bba6e2026969ecfa9bcd8b8682fa60deb0 Oct 31 13:24:50.750508 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.750424 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" event={"ID":"93db7894-3327-4e51-8c0a-81a77124dd7a","Type":"ContainerStarted","Data":"01307333f4cee48d59ecdb2d5c91e8709282036597a2a19303950301633038f2"} Oct 31 13:24:50.752275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.752242 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" event={"ID":"e44283dd-c621-4479-bbd2-b392cfe2a176","Type":"ContainerStarted","Data":"7a16c0ca2dfe92930cb3c4b29d82a5bba6e2026969ecfa9bcd8b8682fa60deb0"} Oct 31 13:24:50.754507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.754053 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" event={"ID":"dcb7b65f-cd03-4643-9a7a-e246ddeafe52","Type":"ContainerStarted","Data":"148f0c190da84b46f408ef4a362bd4644ab79b82e1c68321ca01dc0abdbb12a2"} Oct 31 13:24:50.792336 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.790115 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc"] Oct 31 13:24:50.795561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.795536 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" Oct 31 13:24:50.804128 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.804066 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-5286ea-dockercfg-7xhhv\"" Oct 31 13:24:50.808034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.808007 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc"] Oct 31 13:24:50.855715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.855676 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-5286ea\"" Oct 31 13:24:50.942839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.942184 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr5ch\" (UniqueName: \"kubernetes.io/projected/6343c05e-f79e-4832-9073-cf8e4e97a1f4-kube-api-access-tr5ch\") pod \"payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc\" (UID: \"6343c05e-f79e-4832-9073-cf8e4e97a1f4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" Oct 31 13:24:50.942839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:50.942438 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6343c05e-f79e-4832-9073-cf8e4e97a1f4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc\" (UID: \"6343c05e-f79e-4832-9073-cf8e4e97a1f4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" Oct 31 13:24:51.044913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:51.043890 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6343c05e-f79e-4832-9073-cf8e4e97a1f4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc\" (UID: \"6343c05e-f79e-4832-9073-cf8e4e97a1f4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" Oct 31 13:24:51.044913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:51.043992 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr5ch\" (UniqueName: \"kubernetes.io/projected/6343c05e-f79e-4832-9073-cf8e4e97a1f4-kube-api-access-tr5ch\") pod \"payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc\" (UID: \"6343c05e-f79e-4832-9073-cf8e4e97a1f4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" Oct 31 13:24:51.044913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:51.044827 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6343c05e-f79e-4832-9073-cf8e4e97a1f4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc\" (UID: \"6343c05e-f79e-4832-9073-cf8e4e97a1f4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" Oct 31 13:24:51.064467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:51.064429 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr5ch\" (UniqueName: \"kubernetes.io/projected/6343c05e-f79e-4832-9073-cf8e4e97a1f4-kube-api-access-tr5ch\") pod \"payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc\" (UID: \"6343c05e-f79e-4832-9073-cf8e4e97a1f4\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" Oct 31 13:24:51.124873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:51.124500 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" Oct 31 13:24:51.446899 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:51.446589 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc"] Oct 31 13:24:51.764875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:51.764740 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" event={"ID":"27513f81-9d1f-407b-bca8-86c3d9659bf0","Type":"ContainerStarted","Data":"f79f349a648ce097da929ab5d5934688ada1e93099741d5d68cfebe59baea9af"} Oct 31 13:24:51.764875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:51.764836 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" Oct 31 13:24:51.766739 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:51.766707 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" event={"ID":"6343c05e-f79e-4832-9073-cf8e4e97a1f4","Type":"ContainerStarted","Data":"34249cb3f89824363b6194a0fa6f9b37565fda29c631b205d2c15ae613e34550"} Oct 31 13:24:51.788386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:51.788322 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" podStartSLOduration=1.363709721 podStartE2EDuration="4.788299807s" totalImagesPullingTime="3.424590086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:51.786632908 +0000 UTC m=+1259.932376738" watchObservedRunningTime="2026-10-31 13:24:51.788299807 +0000 UTC m=+1259.934043639" Oct 31 13:24:52.411728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.411698 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:24:52.774430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.774395 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" event={"ID":"dcb7b65f-cd03-4643-9a7a-e246ddeafe52","Type":"ContainerStarted","Data":"95ccff4e9c6f24d96e012fa019641fbfd917631ee713454d350363b7b6b04021"} Oct 31 13:24:52.774649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.774463 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" Oct 31 13:24:52.775977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.775951 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" event={"ID":"93db7894-3327-4e51-8c0a-81a77124dd7a","Type":"ContainerStarted","Data":"2157f51a029454badb0d5b83371428263b64fbd34bd74ce856d6dfef1e9127c5"} Oct 31 13:24:52.776117 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.776027 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" Oct 31 13:24:52.777498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.777471 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" event={"ID":"6cd8e2bd-93c5-4108-a69a-1648d856fede","Type":"ContainerStarted","Data":"77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b"} Oct 31 13:24:52.777646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.777523 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" Oct 31 13:24:52.779062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.779036 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" event={"ID":"6343c05e-f79e-4832-9073-cf8e4e97a1f4","Type":"ContainerStarted","Data":"697275ebcb2d8fef4176736195aec58b64c7e19711ed484c3fbe43c8c21e7dad"} Oct 31 13:24:52.779166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.779101 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" Oct 31 13:24:52.780646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.780625 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" event={"ID":"e44283dd-c621-4479-bbd2-b392cfe2a176","Type":"ContainerStarted","Data":"180b7629250c6670a21424e2b7be4e63d24d6b4d6e276b1118a7d616f1f2584a"} Oct 31 13:24:52.799798 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.799743 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" podStartSLOduration=1.6660178129999998 podStartE2EDuration="3.79972573s" totalImagesPullingTime="2.133707917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:52.796724137 +0000 UTC m=+1260.942467968" watchObservedRunningTime="2026-10-31 13:24:52.79972573 +0000 UTC m=+1260.945469552" Oct 31 13:24:52.849783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.849733 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" podStartSLOduration=2.062088392 podStartE2EDuration="3.849718918s" totalImagesPullingTime="1.787630526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:52.847259875 +0000 UTC m=+1260.993003700" watchObservedRunningTime="2026-10-31 13:24:52.849718918 +0000 UTC m=+1260.995462745" Oct 31 13:24:52.851068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.851034 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" podStartSLOduration=2.485394663 podStartE2EDuration="4.851022635s" totalImagesPullingTime="2.365627972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:52.82663213 +0000 UTC m=+1260.972375956" watchObservedRunningTime="2026-10-31 13:24:52.851022635 +0000 UTC m=+1260.996766464" Oct 31 13:24:52.875717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.875660 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" podStartSLOduration=2.484092195 podStartE2EDuration="2.875641967s" totalImagesPullingTime="391.549772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:52.872975238 +0000 UTC m=+1261.018719063" watchObservedRunningTime="2026-10-31 13:24:52.875641967 +0000 UTC m=+1261.021385797" Oct 31 13:24:52.905568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:52.905514 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" podStartSLOduration=1.830473929 podStartE2EDuration="3.905501188s" totalImagesPullingTime="2.075027259s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:52.903796448 +0000 UTC m=+1261.049540277" watchObservedRunningTime="2026-10-31 13:24:52.905501188 +0000 UTC m=+1261.051245015" Oct 31 13:24:53.787881 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:53.787841 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" Oct 31 13:24:55.087511 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.087471 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b"] Oct 31 13:24:55.092998 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.092976 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.102067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.100734 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-d85d82-serving-cert\"" Oct 31 13:24:55.102067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.100897 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 13:24:55.102067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.100979 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 13:24:55.102067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.101090 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bx4t6\"" Oct 31 13:24:55.102067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.101180 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 13:24:55.102952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.102930 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b"] Oct 31 13:24:55.207737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.207632 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.208061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.208031 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab49081-96ed-405c-80d9-846b102649d7-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.208372 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.208352 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8qbr\" (UniqueName: \"kubernetes.io/projected/4ab49081-96ed-405c-80d9-846b102649d7-kube-api-access-r8qbr\") pod \"maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.208632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.208582 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-metrics-certs\") pod \"maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.309736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.309704 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.309736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.309740 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab49081-96ed-405c-80d9-846b102649d7-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.309951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.309825 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8qbr\" (UniqueName: \"kubernetes.io/projected/4ab49081-96ed-405c-80d9-846b102649d7-kube-api-access-r8qbr\") pod \"maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.309951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.309879 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-metrics-certs\") pod \"maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.310635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.310580 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab49081-96ed-405c-80d9-846b102649d7-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.312255 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.312233 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-maas-api-tls\") pod \"maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.312348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.312270 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-metrics-certs\") pod \"maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.324212 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.324187 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8qbr\" (UniqueName: \"kubernetes.io/projected/4ab49081-96ed-405c-80d9-846b102649d7-kube-api-access-r8qbr\") pod \"maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.412005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.411980 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:55.700781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.700749 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b"] Oct 31 13:24:55.798520 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:55.798487 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" event={"ID":"4ab49081-96ed-405c-80d9-846b102649d7","Type":"ContainerStarted","Data":"09643bcdc2a776b9ac0f8cadd581d864e47bdcab9063b90b032cafb9d5b9ff4d"} Oct 31 13:24:56.003719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:56.003640 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:24:58.820864 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:58.820824 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" event={"ID":"4ab49081-96ed-405c-80d9-846b102649d7","Type":"ContainerStarted","Data":"db43910dc7ea528993d2ade2b37707d28815cefe512a5083c0ce66c4ecc9e70d"} Oct 31 13:24:58.821398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:58.821312 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:24:58.844237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:58.844179 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" podStartSLOduration=1.348652852 podStartE2EDuration="3.844164547s" totalImagesPullingTime="2.495511695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:24:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:24:58.842858048 +0000 UTC m=+1266.988601877" watchObservedRunningTime="2026-10-31 13:24:58.844164547 +0000 UTC m=+1266.989908374" Oct 31 13:24:59.479175 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:24:59.479141 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:25:01.826444 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:25:01.826406 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:25:02.811622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:02.809831 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" Oct 31 13:25:03.229124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.229087 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs"] Oct 31 13:25:03.234823 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.234798 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.265847 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.265810 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6784d6\"/\"default-dockercfg-6gpxf\"" Oct 31 13:25:03.266231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.266205 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6784d6\"/\"simulator-w5-6784d6-kserve-self-signed-certs\"" Oct 31 13:25:03.325654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.325627 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs"] Oct 31 13:25:03.326023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.326000 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6784d6\"/\"openshift-service-ca.crt\"" Oct 31 13:25:03.327750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.327714 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-6784d6\"/\"kube-root-ca.crt\"" Oct 31 13:25:03.428071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.428035 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-model-cache\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.428071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.428075 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-dshm\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.428283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.428147 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tls-certs\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.428283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.428171 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tmp-dir\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.428283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.428192 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgzc9\" (UniqueName: \"kubernetes.io/projected/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-kube-api-access-jgzc9\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.428283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.428245 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-home\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.534546 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.534451 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-home\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.534850 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.534825 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-model-cache\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.534979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.534964 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-dshm\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.535209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.535189 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tls-certs\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.535343 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.535329 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tmp-dir\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.535455 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.535441 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgzc9\" (UniqueName: \"kubernetes.io/projected/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-kube-api-access-jgzc9\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.536331 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.536302 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-home\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.536722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.536701 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-model-cache\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.541845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.541822 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tmp-dir\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.566243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.566203 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tls-certs\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.573617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.568506 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgzc9\" (UniqueName: \"kubernetes.io/projected/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-kube-api-access-jgzc9\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.586196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.586149 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-dshm\") pod \"simulator-w5-6784d6-kserve-5dff7dd579-c9kqs\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.635023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.634991 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:03.819490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.818265 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" Oct 31 13:25:03.819490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.818349 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" Oct 31 13:25:03.819490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.818781 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" Oct 31 13:25:03.819490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:03.819421 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" Oct 31 13:25:04.290624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:04.290024 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs"] Oct 31 13:25:04.818620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:04.818490 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" Oct 31 13:25:04.832940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:04.832915 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 13:25:05.093714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:05.093066 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" event={"ID":"849b0d63-9287-4d49-b4c6-dbdb6af0f37a","Type":"ContainerStarted","Data":"afe9827608151bfdc447788864c79df94f847df42c96447683f23b31095b76a8"} Oct 31 13:25:05.093714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:05.093127 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" event={"ID":"849b0d63-9287-4d49-b4c6-dbdb6af0f37a","Type":"ContainerStarted","Data":"07a0c28bf498ad2b1c91ca3c3823e456c7135f8786011ec8ceaa48c8e4037199"} Oct 31 13:25:05.138460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:05.138398 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podStartSLOduration=2.13838324 podStartE2EDuration="2.13838324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:05.136064057 +0000 UTC m=+1273.281807884" watchObservedRunningTime="2026-10-31 13:25:05.13838324 +0000 UTC m=+1273.284127081" Oct 31 13:25:05.635870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:05.635838 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:06.306091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:06.306059 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:25:08.451101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.451052 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2"] Oct 31 13:25:08.479372 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.479337 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2"] Oct 31 13:25:08.479542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.479453 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.491327 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.487657 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c82140\"/\"simulator-w3-c82140-kserve-self-signed-certs\"" Oct 31 13:25:08.491327 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.488231 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c82140\"/\"openshift-service-ca.crt\"" Oct 31 13:25:08.491327 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.489252 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c82140\"/\"kube-root-ca.crt\"" Oct 31 13:25:08.491327 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.490584 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c82140\"/\"default-dockercfg-mzt5g\"" Oct 31 13:25:08.560966 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.560926 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-dshm\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.561152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.561068 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-model-cache\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.561152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.561127 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp24g\" (UniqueName: \"kubernetes.io/projected/dccc053f-a13f-496e-8c6e-781f41397b58-kube-api-access-tp24g\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.561293 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.561266 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccc053f-a13f-496e-8c6e-781f41397b58-tls-certs\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.561553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.561382 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-home\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.561553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.561441 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-tmp-dir\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.665655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.662784 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-dshm\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.665655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.662873 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-model-cache\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.665655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.662923 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp24g\" (UniqueName: \"kubernetes.io/projected/dccc053f-a13f-496e-8c6e-781f41397b58-kube-api-access-tp24g\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.665655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.663004 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccc053f-a13f-496e-8c6e-781f41397b58-tls-certs\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.665655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.663072 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-home\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.665655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.663116 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-tmp-dir\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.665655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.663649 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-tmp-dir\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.665655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.665024 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-model-cache\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.665655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.665265 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-home\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.679453 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.676273 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-dshm\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.679453 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.677645 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccc053f-a13f-496e-8c6e-781f41397b58-tls-certs\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.686898 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.686869 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp24g\" (UniqueName: \"kubernetes.io/projected/dccc053f-a13f-496e-8c6e-781f41397b58-kube-api-access-tp24g\") pod \"simulator-w3-c82140-kserve-5867bc8964-sqxn2\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:08.832652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:08.832542 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:09.158164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:09.158128 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2"] Oct 31 13:25:09.159376 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:25:09.159345 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddccc053f_a13f_496e_8c6e_781f41397b58.slice/crio-41f767cebc0e34d589b024bb3f00d006166e505bc6ab7177b4630eefddbeb831 WatchSource:0}: Error finding container 41f767cebc0e34d589b024bb3f00d006166e505bc6ab7177b4630eefddbeb831: Status 404 returned error can't find the container with id 41f767cebc0e34d589b024bb3f00d006166e505bc6ab7177b4630eefddbeb831 Oct 31 13:25:10.127975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:10.127931 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" event={"ID":"dccc053f-a13f-496e-8c6e-781f41397b58","Type":"ContainerStarted","Data":"8287b0a564d9f1f2384fc4d86efe6ff6a1e4ca093e0bdd1a2baccfc8fcbd1be0"} Oct 31 13:25:10.127975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:10.127980 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" event={"ID":"dccc053f-a13f-496e-8c6e-781f41397b58","Type":"ContainerStarted","Data":"41f767cebc0e34d589b024bb3f00d006166e505bc6ab7177b4630eefddbeb831"} Oct 31 13:25:11.444837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:10.832916 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:12.139157 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.139094 5532 generic.go:356] "Generic (PLEG): container finished" podID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerID="2735d3ea10d10be65c87f9c9ec36ca13c4c607e7d15a4bde348cdc030d896f9b" exitCode=0 Oct 31 13:25:12.139308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.139163 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" event={"ID":"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4","Type":"ContainerDied","Data":"2735d3ea10d10be65c87f9c9ec36ca13c4c607e7d15a4bde348cdc030d896f9b"} Oct 31 13:25:12.139625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.139569 5532 scope.go:122] "RemoveContainer" containerID="2735d3ea10d10be65c87f9c9ec36ca13c4c607e7d15a4bde348cdc030d896f9b" Oct 31 13:25:12.140835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.140816 5532 generic.go:356] "Generic (PLEG): container finished" podID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerID="a6db087353b60087ed320ed226d37351b82d2bda2c61292278f84a9a6102f5b3" exitCode=0 Oct 31 13:25:12.140937 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.140886 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" event={"ID":"928c06ae-3fb5-4d89-aa1f-0da90832da40","Type":"ContainerDied","Data":"a6db087353b60087ed320ed226d37351b82d2bda2c61292278f84a9a6102f5b3"} Oct 31 13:25:12.141021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.140992 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:25:12.141081 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.141065 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:25:12.141138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.141102 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:25:12.141289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.141271 5532 scope.go:122] "RemoveContainer" containerID="a6db087353b60087ed320ed226d37351b82d2bda2c61292278f84a9a6102f5b3" Oct 31 13:25:12.142569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.142543 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:25:12.142723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.142640 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:25:12.142723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.142693 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:25:12.142843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.142730 5532 generic.go:356] "Generic (PLEG): container finished" podID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerID="dd02214c26380e2ef79dacdbf9badce4d57811fff29ca0ce19a075348c5146b4" exitCode=0 Oct 31 13:25:12.142843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.142829 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" event={"ID":"8a7ef42d-5b18-414a-a0ff-492cc511c0db","Type":"ContainerDied","Data":"dd02214c26380e2ef79dacdbf9badce4d57811fff29ca0ce19a075348c5146b4"} Oct 31 13:25:12.143317 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.143301 5532 scope.go:122] "RemoveContainer" containerID="dd02214c26380e2ef79dacdbf9badce4d57811fff29ca0ce19a075348c5146b4" Oct 31 13:25:12.144415 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.144388 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:25:12.144539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.144454 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:25:12.144588 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.144545 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:25:12.172997 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.172951 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podStartSLOduration=4.172937533 podStartE2EDuration="4.172937533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:10.149313715 +0000 UTC m=+1278.295057534" watchObservedRunningTime="2026-10-31 13:25:12.172937533 +0000 UTC m=+1280.318681360" Oct 31 13:25:12.420824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:12.420785 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:25:13.210617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.199969 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" event={"ID":"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4","Type":"ContainerStarted","Data":"bb552b3a689ad9b0102b281738bbfb59b6a006990466e803857fe92c6d429510"} Oct 31 13:25:13.210617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.203411 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" event={"ID":"928c06ae-3fb5-4d89-aa1f-0da90832da40","Type":"ContainerStarted","Data":"6b787d58998e63da4ce9230b0a58159346851e8dd2b2023522681771d04dd75a"} Oct 31 13:25:13.223447 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.223367 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" event={"ID":"8a7ef42d-5b18-414a-a0ff-492cc511c0db","Type":"ContainerStarted","Data":"a2b7912fbf65c0403d50a81b01fe037857d9412d10d8b681789cb5788a102c52"} Oct 31 13:25:13.337451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.337420 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:25:13.338277 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.338255 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 13:25:13.338334 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.338314 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 13:25:13.578682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.577936 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:25:13.594628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.588896 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 13:25:13.594628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.588958 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 13:25:13.637267 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.635973 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:13.734170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.734137 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:13.833227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.833138 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:25:13.834172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.834152 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 13:25:13.834251 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.834194 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 13:25:13.983620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:13.983199 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j"] Oct 31 13:25:14.032791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.032404 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j"] Oct 31 13:25:14.032791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.032581 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.092674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.037871 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aabeef\"/\"simulator-w6-aabeef-kserve-self-signed-certs\"" Oct 31 13:25:14.092674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.079881 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-aabeef\"/\"default-dockercfg-9t4bh\"" Oct 31 13:25:14.092674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.079983 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aabeef\"/\"openshift-service-ca.crt\"" Oct 31 13:25:14.092674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.080083 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-aabeef\"/\"kube-root-ca.crt\"" Oct 31 13:25:14.137069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.136380 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tmp-dir\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.137069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.136414 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-home\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.137069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.136580 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mftx7\" (UniqueName: \"kubernetes.io/projected/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-kube-api-access-mftx7\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.137069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.136650 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-dshm\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.137069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.136710 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-model-cache\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.137069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.136915 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tls-certs\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.237823 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.237777 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-dshm\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.238473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.238448 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-model-cache\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.238884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.238864 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tls-certs\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.239065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.239048 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tmp-dir\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.239362 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.239343 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-home\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.245380 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.244972 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mftx7\" (UniqueName: \"kubernetes.io/projected/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-kube-api-access-mftx7\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.261320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.243993 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-home\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.261515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.244317 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-model-cache\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.261655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.239856 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tmp-dir\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.280629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.273930 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tls-certs\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.294615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.289755 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-dshm\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.294615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.290431 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mftx7\" (UniqueName: \"kubernetes.io/projected/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-kube-api-access-mftx7\") pod \"simulator-w6-aabeef-kserve-698f45dd6f-8fh7j\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.329984 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.329703 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:25:14.337194 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.337169 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 13:25:14.337382 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.337215 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 13:25:14.400668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.396792 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:14.601618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.599097 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 13:25:14.601618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.599158 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 13:25:14.834801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.833994 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 13:25:14.834801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.834051 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 13:25:14.873620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:14.873558 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j"] Oct 31 13:25:15.314619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:15.311690 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" event={"ID":"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7","Type":"ContainerStarted","Data":"8cc250dca7c653594d074a1ba0ff8b8ac53961d39d7899f965c2bd9084a9291f"} Oct 31 13:25:15.314619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:15.311761 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" event={"ID":"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7","Type":"ContainerStarted","Data":"6e42b645e3b1da55e9f23b8c8a3bb7b9d64ebdb67aa718b9e58f4d2cb1ddfd39"} Oct 31 13:25:15.337026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:15.336967 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 13:25:15.337844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:15.337317 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 13:25:15.420870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:15.420835 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:15.459619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:15.448910 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podStartSLOduration=2.44888821 podStartE2EDuration="2.44888821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:15.339398311 +0000 UTC m=+1283.485142141" watchObservedRunningTime="2026-10-31 13:25:15.44888821 +0000 UTC m=+1283.594632040" Oct 31 13:25:15.583066 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:15.583024 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 13:25:15.583377 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:15.583336 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 13:25:15.679815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:15.679702 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:25:15.680632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:15.680587 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:25:15.833638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:15.833576 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 13:25:15.833809 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:15.833660 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 13:25:16.359139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:16.359101 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 13:25:16.359768 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:16.359733 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 13:25:16.584108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:16.584070 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 13:25:16.584387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:16.584359 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 13:25:16.674618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:16.673731 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:25:16.833041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:16.833000 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 13:25:16.833332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:16.833294 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 13:25:17.349619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:17.337960 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 13:25:17.349619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:17.338021 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 13:25:17.582556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:17.579170 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Oct 31 13:25:17.582556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:17.579234 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Oct 31 13:25:17.835630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:17.834723 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Oct 31 13:25:17.835630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:17.834789 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Oct 31 13:25:18.374714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:18.374128 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Oct 31 13:25:18.374714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:18.374203 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Oct 31 13:25:18.833475 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:18.833434 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:18.839590 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:18.839564 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:25:18.840626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:18.840578 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:25:18.853101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:18.853076 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:18.902354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:18.902309 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:25:18.905804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:18.905780 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:25:19.342881 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:19.342853 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:25:19.344172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:19.344151 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:25:19.354207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:19.354186 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:25:21.407267 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.407229 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd"] Oct 31 13:25:21.412772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.412748 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.429540 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.429517 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d85d82\"/\"default-dockercfg-jzcf5\"" Oct 31 13:25:21.429748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.429663 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d85d82\"/\"simulator-w1-d85d82-kserve-self-signed-certs\"" Oct 31 13:25:21.431208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.431188 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d85d82\"/\"kube-root-ca.crt\"" Oct 31 13:25:21.431329 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.431310 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-d85d82\"/\"openshift-service-ca.crt\"" Oct 31 13:25:21.433818 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.433801 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd"] Oct 31 13:25:21.477737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.477701 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-model-cache\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.477921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.477744 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-home\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.477921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.477770 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-tmp-dir\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.477921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.477837 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vwhg\" (UniqueName: \"kubernetes.io/projected/69038604-7198-4e89-be50-1a155aaa7c75-kube-api-access-5vwhg\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.477921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.477888 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-dshm\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.478082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.477926 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69038604-7198-4e89-be50-1a155aaa7c75-tls-certs\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.578610 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.578569 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-model-cache\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.578784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.578662 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-home\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.578784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.578701 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-tmp-dir\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.578784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.578728 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vwhg\" (UniqueName: \"kubernetes.io/projected/69038604-7198-4e89-be50-1a155aaa7c75-kube-api-access-5vwhg\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.578784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.578763 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-dshm\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.578982 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.578802 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69038604-7198-4e89-be50-1a155aaa7c75-tls-certs\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.579061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.579038 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-model-cache\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.579116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.579058 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-home\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.579160 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.579136 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-tmp-dir\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.581113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.581084 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-dshm\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.581323 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.581302 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69038604-7198-4e89-be50-1a155aaa7c75-tls-certs\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.594203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.594170 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vwhg\" (UniqueName: \"kubernetes.io/projected/69038604-7198-4e89-be50-1a155aaa7c75-kube-api-access-5vwhg\") pod \"simulator-w1-d85d82-kserve-6dddf5465-8nntd\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:21.725066 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:21.724976 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:22.004112 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:25:22.004083 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69038604_7198_4e89_be50_1a155aaa7c75.slice/crio-2ff0ab847721014708ecd1500baca02cf5c1ab77fb83787937f1809becd5e20d WatchSource:0}: Error finding container 2ff0ab847721014708ecd1500baca02cf5c1ab77fb83787937f1809becd5e20d: Status 404 returned error can't find the container with id 2ff0ab847721014708ecd1500baca02cf5c1ab77fb83787937f1809becd5e20d Oct 31 13:25:22.004236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:22.004174 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd"] Oct 31 13:25:22.360017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:22.359928 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" event={"ID":"69038604-7198-4e89-be50-1a155aaa7c75","Type":"ContainerStarted","Data":"c2fad5b6b956521fc75065e7de8355394756b996e04f047b0bbe60155bc84639"} Oct 31 13:25:22.360017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:22.359978 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" event={"ID":"69038604-7198-4e89-be50-1a155aaa7c75","Type":"ContainerStarted","Data":"2ff0ab847721014708ecd1500baca02cf5c1ab77fb83787937f1809becd5e20d"} Oct 31 13:25:22.385277 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:22.385222 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podStartSLOduration=1.385209278 podStartE2EDuration="1.385209278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:22.382843228 +0000 UTC m=+1290.528587057" watchObservedRunningTime="2026-10-31 13:25:22.385209278 +0000 UTC m=+1290.530953106" Oct 31 13:25:22.725588 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:22.725552 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:24.203943 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:24.203907 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:25:24.417569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:24.417534 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:24.430468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:24.430434 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:25.388162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:25.388130 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:25:31.725756 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:31.725717 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:31.738544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:31.738513 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:31.762772 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:25:31.762740 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:25:32.293627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.293577 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz"] Oct 31 13:25:32.297603 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.297577 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.303937 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.303913 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-6784d6\"/\"premium-w5-6784d6-kserve-self-signed-certs\"" Oct 31 13:25:32.313360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.313331 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz"] Oct 31 13:25:32.403074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.403035 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctsm4\" (UniqueName: \"kubernetes.io/projected/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-kube-api-access-ctsm4\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.403253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.403157 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-model-cache\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.403253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.403230 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-dshm\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.403376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.403316 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-home\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.403433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.403398 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tmp-dir\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.403486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.403457 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tls-certs\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.431692 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.431666 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:25:32.505359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.505320 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tls-certs\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.506181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.506155 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctsm4\" (UniqueName: \"kubernetes.io/projected/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-kube-api-access-ctsm4\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.507027 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.506996 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-model-cache\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.507163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.507142 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-dshm\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.507236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.507191 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-home\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.507236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.507225 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tmp-dir\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.507553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.507531 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tmp-dir\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.508301 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.507956 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-home\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.508301 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.508260 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-model-cache\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.511721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.511676 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tls-certs\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.513891 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.513851 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-dshm\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.525340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.525306 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctsm4\" (UniqueName: \"kubernetes.io/projected/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-kube-api-access-ctsm4\") pod \"premium-w5-6784d6-kserve-75dc9b6d46-gcbwz\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.611221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.611129 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:32.897283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:32.897075 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz"] Oct 31 13:25:33.427531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:33.427496 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" event={"ID":"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8","Type":"ContainerStarted","Data":"204fcbc9dbddb55792c9be72e3d4cdce02c7c801fa647b7db1e68737de0507ef"} Oct 31 13:25:33.427752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:33.427541 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" event={"ID":"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8","Type":"ContainerStarted","Data":"70ab02fa043a5d0ff5da8a708a9e07195675e7cdda2a00da458b2ee7a8d70e34"} Oct 31 13:25:33.450092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:33.450040 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podStartSLOduration=1.4500256870000001 podStartE2EDuration="1.450025687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:33.447849113 +0000 UTC m=+1301.593592944" watchObservedRunningTime="2026-10-31 13:25:33.450025687 +0000 UTC m=+1301.595769515" Oct 31 13:25:33.612198 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:33.612166 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:37.400924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:37.400889 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:25:42.611643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:42.611611 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:42.624259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:42.624236 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:43.497631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:43.497586 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:25:47.078902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.078871 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q"] Oct 31 13:25:47.082974 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.082956 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.095155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.095128 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-d85d82\"/\"premium-w1-d85d82-kserve-self-signed-certs\"" Oct 31 13:25:47.097185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.097161 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q"] Oct 31 13:25:47.178270 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.178237 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh7l5\" (UniqueName: \"kubernetes.io/projected/1e02db96-5a0a-4bca-812e-af51948d7ab0-kube-api-access-sh7l5\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.178473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.178305 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e02db96-5a0a-4bca-812e-af51948d7ab0-tls-certs\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.178473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.178393 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-dshm\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.178473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.178424 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-model-cache\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.178473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.178452 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-tmp-dir\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.178735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.178543 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-home\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.279305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.279269 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e02db96-5a0a-4bca-812e-af51948d7ab0-tls-certs\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.279492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.279329 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-dshm\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.279492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.279356 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-model-cache\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.279492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.279384 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-tmp-dir\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.279492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.279405 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-home\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.279492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.279479 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh7l5\" (UniqueName: \"kubernetes.io/projected/1e02db96-5a0a-4bca-812e-af51948d7ab0-kube-api-access-sh7l5\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.279846 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.279823 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-model-cache\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.279927 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.279866 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-home\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.279927 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.279913 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-tmp-dir\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.281667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.281641 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-dshm\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.281903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.281884 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e02db96-5a0a-4bca-812e-af51948d7ab0-tls-certs\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.298316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.298294 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh7l5\" (UniqueName: \"kubernetes.io/projected/1e02db96-5a0a-4bca-812e-af51948d7ab0-kube-api-access-sh7l5\") pod \"premium-w1-d85d82-kserve-d5d69db44-wt98q\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.396216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.396182 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:47.677985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:47.677958 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q"] Oct 31 13:25:47.679641 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:25:47.679617 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e02db96_5a0a_4bca_812e_af51948d7ab0.slice/crio-779b2bbf105441b2d460151061cc4f08f0933d90e3e49e6faf763d81b41113c2 WatchSource:0}: Error finding container 779b2bbf105441b2d460151061cc4f08f0933d90e3e49e6faf763d81b41113c2: Status 404 returned error can't find the container with id 779b2bbf105441b2d460151061cc4f08f0933d90e3e49e6faf763d81b41113c2 Oct 31 13:25:48.522643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:48.522578 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" event={"ID":"1e02db96-5a0a-4bca-812e-af51948d7ab0","Type":"ContainerStarted","Data":"caf649ca72c724aa6644224ec71c71dfc184eb0db2943a1a2b94c24ecba5eecb"} Oct 31 13:25:48.522643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:48.522647 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" event={"ID":"1e02db96-5a0a-4bca-812e-af51948d7ab0","Type":"ContainerStarted","Data":"779b2bbf105441b2d460151061cc4f08f0933d90e3e49e6faf763d81b41113c2"} Oct 31 13:25:48.548153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:48.548107 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podStartSLOduration=1.548094856 podStartE2EDuration="1.548094856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:25:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:25:48.547821416 +0000 UTC m=+1316.693565243" watchObservedRunningTime="2026-10-31 13:25:48.548094856 +0000 UTC m=+1316.693838714" Oct 31 13:25:49.397181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:49.397149 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:53.403543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:53.403509 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:25:57.396579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:57.396541 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:57.409172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:57.409151 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:25:57.583627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:25:57.583586 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:26:01.760486 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:26:01.760446 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:26:16.728611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:16.728562 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:26:28.125468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.125435 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5"] Oct 31 13:26:28.130180 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.130159 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.138699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.138672 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-6784d6-openshift-default-dockercfg-xbv85\"" Oct 31 13:26:28.144732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.144706 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5"] Oct 31 13:26:28.275978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.275947 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.276164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.276061 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.276164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.276099 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.276290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.276183 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.276290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.276243 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.276362 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.276316 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.276405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.276375 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.276445 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.276407 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.276445 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.276431 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62rgz\" (UniqueName: \"kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-kube-api-access-62rgz\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.377551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.377471 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.377551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.377549 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.377801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.377579 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.377801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.377630 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.377801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.377679 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.377801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.377712 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.377801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.377772 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.378059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.377822 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.378059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.377849 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62rgz\" (UniqueName: \"kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-kube-api-access-62rgz\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.378059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.377937 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.378211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.378170 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.378272 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.378206 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.378326 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.378310 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.378455 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.378433 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.380624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.380582 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.380770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.380673 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.392211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.392187 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62rgz\" (UniqueName: \"kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-kube-api-access-62rgz\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.392348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.392249 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.443937 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.443913 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:28.741340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.741165 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5"] Oct 31 13:26:28.748740 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:26:28.748707 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbf59a63_8f9e_4fb5_acb3_2ca0cf6fba23.slice/crio-7a3ab1d34c212e7fc0c7a01d295b21856fdb12607a51e6076c8ff6c3492d5fed WatchSource:0}: Error finding container 7a3ab1d34c212e7fc0c7a01d295b21856fdb12607a51e6076c8ff6c3492d5fed: Status 404 returned error can't find the container with id 7a3ab1d34c212e7fc0c7a01d295b21856fdb12607a51e6076c8ff6c3492d5fed Oct 31 13:26:28.751097 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.751051 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:26:28.751205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.751149 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:26:28.751205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.751191 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:26:28.767728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:28.767701 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" event={"ID":"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23","Type":"ContainerStarted","Data":"7a3ab1d34c212e7fc0c7a01d295b21856fdb12607a51e6076c8ff6c3492d5fed"} Oct 31 13:26:29.774355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:29.774269 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" event={"ID":"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23","Type":"ContainerStarted","Data":"284af228b9b6d4e7dec60804bc214244db3a6ad942bbb5e377bea114cdfb8f0d"} Oct 31 13:26:29.802154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:29.802107 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" podStartSLOduration=1.802093425 podStartE2EDuration="1.802093425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:29.801283508 +0000 UTC m=+1357.947027336" watchObservedRunningTime="2026-10-31 13:26:29.802093425 +0000 UTC m=+1357.947837316" Oct 31 13:26:30.459619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:30.446588 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:30.459619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:30.446790 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 13:26:30.459619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:30.446843 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" podUID="dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 13:26:31.457628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:31.456913 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 13:26:31.457628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:31.456986 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" podUID="dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 13:26:31.784167 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:26:31.783855 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:26:32.450630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:32.449892 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 13:26:32.450630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:32.449949 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" podUID="dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 13:26:33.457618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:33.452941 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 13:26:33.457618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:33.453002 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" podUID="dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 13:26:34.464051 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:34.464009 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Oct 31 13:26:34.464462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:34.464068 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" podUID="dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Oct 31 13:26:35.448521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.448491 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:35.448865 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.448845 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:35.449533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.449514 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 13:26:35.460570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.460539 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq"] Oct 31 13:26:35.464922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.464901 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" Oct 31 13:26:35.473152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.473025 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:26:35.473152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.473127 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:26:35.475259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.475238 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq"] Oct 31 13:26:35.590727 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.590693 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cbpt\" (UniqueName: \"kubernetes.io/projected/a99e9d6a-eeec-490b-8d43-86199491b259-kube-api-access-4cbpt\") pod \"maas-api-revoke-sub-52c82151-62hpq\" (UID: \"a99e9d6a-eeec-490b-8d43-86199491b259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" Oct 31 13:26:35.590940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.590911 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52c82151-62hpq\" (UID: \"a99e9d6a-eeec-490b-8d43-86199491b259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" Oct 31 13:26:35.591032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.591002 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52c82151-62hpq\" (UID: \"a99e9d6a-eeec-490b-8d43-86199491b259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" Oct 31 13:26:35.692456 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.692422 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52c82151-62hpq\" (UID: \"a99e9d6a-eeec-490b-8d43-86199491b259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" Oct 31 13:26:35.692456 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.692465 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52c82151-62hpq\" (UID: \"a99e9d6a-eeec-490b-8d43-86199491b259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" Oct 31 13:26:35.692763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.692547 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cbpt\" (UniqueName: \"kubernetes.io/projected/a99e9d6a-eeec-490b-8d43-86199491b259-kube-api-access-4cbpt\") pod \"maas-api-revoke-sub-52c82151-62hpq\" (UID: \"a99e9d6a-eeec-490b-8d43-86199491b259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" Oct 31 13:26:35.693294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.693265 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52c82151-62hpq\" (UID: \"a99e9d6a-eeec-490b-8d43-86199491b259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" Oct 31 13:26:35.695076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.695046 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52c82151-62hpq\" (UID: \"a99e9d6a-eeec-490b-8d43-86199491b259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" Oct 31 13:26:35.712342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.712273 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cbpt\" (UniqueName: \"kubernetes.io/projected/a99e9d6a-eeec-490b-8d43-86199491b259-kube-api-access-4cbpt\") pod \"maas-api-revoke-sub-52c82151-62hpq\" (UID: \"a99e9d6a-eeec-490b-8d43-86199491b259\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" Oct 31 13:26:35.780737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:35.780700 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" Oct 31 13:26:36.070153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:36.070110 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq"] Oct 31 13:26:36.071930 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:26:36.071905 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99e9d6a_eeec_490b_8d43_86199491b259.slice/crio-78f2eca491ee7fa7b1dc5850214c2a5fcb4d83cb22e05803daabf5afba32b4f7 WatchSource:0}: Error finding container 78f2eca491ee7fa7b1dc5850214c2a5fcb4d83cb22e05803daabf5afba32b4f7: Status 404 returned error can't find the container with id 78f2eca491ee7fa7b1dc5850214c2a5fcb4d83cb22e05803daabf5afba32b4f7 Oct 31 13:26:36.074267 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:36.074244 5532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:26:36.824987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:36.824947 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" event={"ID":"a99e9d6a-eeec-490b-8d43-86199491b259","Type":"ContainerStarted","Data":"78f2eca491ee7fa7b1dc5850214c2a5fcb4d83cb22e05803daabf5afba32b4f7"} Oct 31 13:26:39.847760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:39.847720 5532 generic.go:356] "Generic (PLEG): container finished" podID="a99e9d6a-eeec-490b-8d43-86199491b259" containerID="dd0d2558d2a22bdcaea06f01d3a911c5c7e9c8aae340242dc56590906da805d0" exitCode=0 Oct 31 13:26:39.848136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:39.847760 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" event={"ID":"a99e9d6a-eeec-490b-8d43-86199491b259","Type":"ContainerDied","Data":"dd0d2558d2a22bdcaea06f01d3a911c5c7e9c8aae340242dc56590906da805d0"} Oct 31 13:26:41.004473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:41.004448 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" Oct 31 13:26:41.050730 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:41.050692 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99e9d6a-eeec-490b-8d43-86199491b259-kube-api-access-4cbpt\" (UniqueName: \"kubernetes.io/projected/a99e9d6a-eeec-490b-8d43-86199491b259-kube-api-access-4cbpt\") pod \"a99e9d6a-eeec-490b-8d43-86199491b259\" (UID: \"a99e9d6a-eeec-490b-8d43-86199491b259\") " Oct 31 13:26:41.050913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:41.050824 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-xks-ca\") pod \"a99e9d6a-eeec-490b-8d43-86199491b259\" (UID: \"a99e9d6a-eeec-490b-8d43-86199491b259\") " Oct 31 13:26:41.051010 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:41.050990 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-openshift-ca\") pod \"a99e9d6a-eeec-490b-8d43-86199491b259\" (UID: \"a99e9d6a-eeec-490b-8d43-86199491b259\") " Oct 31 13:26:41.051406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:41.051377 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-openshift-ca" pod "a99e9d6a-eeec-490b-8d43-86199491b259" (UID: "a99e9d6a-eeec-490b-8d43-86199491b259"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:41.051556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:41.051536 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:41.052986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:41.052959 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-xks-ca" pod "a99e9d6a-eeec-490b-8d43-86199491b259" (UID: "a99e9d6a-eeec-490b-8d43-86199491b259"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:26:41.053087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:41.053030 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99e9d6a-eeec-490b-8d43-86199491b259-kube-api-access-4cbpt" pod "a99e9d6a-eeec-490b-8d43-86199491b259" (UID: "a99e9d6a-eeec-490b-8d43-86199491b259"). InnerVolumeSpecName "kube-api-access-4cbpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:41.153202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:41.153175 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cbpt\" (UniqueName: \"kubernetes.io/projected/a99e9d6a-eeec-490b-8d43-86199491b259-kube-api-access-4cbpt\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:41.153202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:41.153202 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a99e9d6a-eeec-490b-8d43-86199491b259-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:41.861137 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:41.861107 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" Oct 31 13:26:41.861310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:41.861143 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq" event={"ID":"a99e9d6a-eeec-490b-8d43-86199491b259","Type":"ContainerDied","Data":"78f2eca491ee7fa7b1dc5850214c2a5fcb4d83cb22e05803daabf5afba32b4f7"} Oct 31 13:26:41.861310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:41.861190 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78f2eca491ee7fa7b1dc5850214c2a5fcb4d83cb22e05803daabf5afba32b4f7" Oct 31 13:26:46.406096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.406063 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj"] Oct 31 13:26:46.406639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.406622 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a99e9d6a-eeec-490b-8d43-86199491b259" containerName="revoke-keys" Oct 31 13:26:46.406699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.406640 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99e9d6a-eeec-490b-8d43-86199491b259" containerName="revoke-keys" Oct 31 13:26:46.406764 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.406755 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99e9d6a-eeec-490b-8d43-86199491b259" containerName="revoke-keys" Oct 31 13:26:46.410552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.410531 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" Oct 31 13:26:46.418906 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.418886 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:26:46.419050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.418927 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:26:46.420466 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.420444 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj"] Oct 31 13:26:46.513551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.513522 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb59233-xnzvj\" (UID: \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" Oct 31 13:26:46.513745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.513726 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb59233-xnzvj\" (UID: \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" Oct 31 13:26:46.513818 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.513767 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wppxl\" (UniqueName: \"kubernetes.io/projected/56ea5bdf-96e9-4d3c-8d85-98494a327a40-kube-api-access-wppxl\") pod \"maas-api-revoke-sub-5fb59233-xnzvj\" (UID: \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" Oct 31 13:26:46.615397 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.615359 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb59233-xnzvj\" (UID: \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" Oct 31 13:26:46.615655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.615548 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb59233-xnzvj\" (UID: \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" Oct 31 13:26:46.615655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.615625 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wppxl\" (UniqueName: \"kubernetes.io/projected/56ea5bdf-96e9-4d3c-8d85-98494a327a40-kube-api-access-wppxl\") pod \"maas-api-revoke-sub-5fb59233-xnzvj\" (UID: \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" Oct 31 13:26:46.616375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.616352 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5fb59233-xnzvj\" (UID: \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" Oct 31 13:26:46.617800 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.617773 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5fb59233-xnzvj\" (UID: \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" Oct 31 13:26:46.631179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.631150 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wppxl\" (UniqueName: \"kubernetes.io/projected/56ea5bdf-96e9-4d3c-8d85-98494a327a40-kube-api-access-wppxl\") pod \"maas-api-revoke-sub-5fb59233-xnzvj\" (UID: \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" Oct 31 13:26:46.726442 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:46.726348 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" Oct 31 13:26:47.021657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:47.021629 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj"] Oct 31 13:26:47.023703 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:26:47.023671 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ea5bdf_96e9_4d3c_8d85_98494a327a40.slice/crio-50b056be003dc985f5aae7fd149a81cf80114b75e64c80def0ca5c0c6c638cc5 WatchSource:0}: Error finding container 50b056be003dc985f5aae7fd149a81cf80114b75e64c80def0ca5c0c6c638cc5: Status 404 returned error can't find the container with id 50b056be003dc985f5aae7fd149a81cf80114b75e64c80def0ca5c0c6c638cc5 Oct 31 13:26:47.897148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:47.897107 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" event={"ID":"56ea5bdf-96e9-4d3c-8d85-98494a327a40","Type":"ContainerStarted","Data":"e126f3d454a5890579fe0e6d98686c09f026b207260da7a06c5aad2c04b2ba50"} Oct 31 13:26:47.897680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:47.897167 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" event={"ID":"56ea5bdf-96e9-4d3c-8d85-98494a327a40","Type":"ContainerStarted","Data":"50b056be003dc985f5aae7fd149a81cf80114b75e64c80def0ca5c0c6c638cc5"} Oct 31 13:26:47.920366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:47.920321 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" podStartSLOduration=1.920310032 podStartE2EDuration="1.920310032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:47.919153307 +0000 UTC m=+1376.064897135" watchObservedRunningTime="2026-10-31 13:26:47.920310032 +0000 UTC m=+1376.066053860" Oct 31 13:26:48.904779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:48.904744 5532 generic.go:356] "Generic (PLEG): container finished" podID="56ea5bdf-96e9-4d3c-8d85-98494a327a40" containerID="e126f3d454a5890579fe0e6d98686c09f026b207260da7a06c5aad2c04b2ba50" exitCode=0 Oct 31 13:26:48.905166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:48.904816 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" event={"ID":"56ea5bdf-96e9-4d3c-8d85-98494a327a40","Type":"ContainerDied","Data":"e126f3d454a5890579fe0e6d98686c09f026b207260da7a06c5aad2c04b2ba50"} Oct 31 13:26:50.065052 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.065028 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" Oct 31 13:26:50.158824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.158793 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ea5bdf-96e9-4d3c-8d85-98494a327a40-kube-api-access-wppxl\" (UniqueName: \"kubernetes.io/projected/56ea5bdf-96e9-4d3c-8d85-98494a327a40-kube-api-access-wppxl\") pod \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\" (UID: \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\") " Oct 31 13:26:50.158984 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.158952 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-openshift-ca\") pod \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\" (UID: \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\") " Oct 31 13:26:50.159058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.159044 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-xks-ca\") pod \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\" (UID: \"56ea5bdf-96e9-4d3c-8d85-98494a327a40\") " Oct 31 13:26:50.159425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.159396 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-openshift-ca" pod "56ea5bdf-96e9-4d3c-8d85-98494a327a40" (UID: "56ea5bdf-96e9-4d3c-8d85-98494a327a40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:50.159558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.159482 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:50.160942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.160916 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ea5bdf-96e9-4d3c-8d85-98494a327a40-kube-api-access-wppxl" pod "56ea5bdf-96e9-4d3c-8d85-98494a327a40" (UID: "56ea5bdf-96e9-4d3c-8d85-98494a327a40"). InnerVolumeSpecName "kube-api-access-wppxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:50.161196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.161173 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-xks-ca" pod "56ea5bdf-96e9-4d3c-8d85-98494a327a40" (UID: "56ea5bdf-96e9-4d3c-8d85-98494a327a40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:26:50.260276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.260244 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ea5bdf-96e9-4d3c-8d85-98494a327a40-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:50.260276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.260273 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wppxl\" (UniqueName: \"kubernetes.io/projected/56ea5bdf-96e9-4d3c-8d85-98494a327a40-kube-api-access-wppxl\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:50.750554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.750521 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6"] Oct 31 13:26:50.751109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.751095 5532 cpu_manager.go:414] "containerMap: removing container" podUID="56ea5bdf-96e9-4d3c-8d85-98494a327a40" containerName="revoke-keys" Oct 31 13:26:50.751161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.751111 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ea5bdf-96e9-4d3c-8d85-98494a327a40" containerName="revoke-keys" Oct 31 13:26:50.751213 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.751204 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ea5bdf-96e9-4d3c-8d85-98494a327a40" containerName="revoke-keys" Oct 31 13:26:50.754561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.754543 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" Oct 31 13:26:50.764339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.764312 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6"] Oct 31 13:26:50.868876 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.868838 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97538cf4-g2wv6\" (UID: \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" Oct 31 13:26:50.869047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.868899 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97538cf4-g2wv6\" (UID: \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" Oct 31 13:26:50.869047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.869019 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsfh5\" (UniqueName: \"kubernetes.io/projected/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-kube-api-access-jsfh5\") pod \"maas-api-revoke-sub-97538cf4-g2wv6\" (UID: \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" Oct 31 13:26:50.919073 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.919035 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" event={"ID":"56ea5bdf-96e9-4d3c-8d85-98494a327a40","Type":"ContainerDied","Data":"50b056be003dc985f5aae7fd149a81cf80114b75e64c80def0ca5c0c6c638cc5"} Oct 31 13:26:50.919245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.919077 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj" Oct 31 13:26:50.919245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.919095 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50b056be003dc985f5aae7fd149a81cf80114b75e64c80def0ca5c0c6c638cc5" Oct 31 13:26:50.970769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.970737 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97538cf4-g2wv6\" (UID: \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" Oct 31 13:26:50.970915 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.970811 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97538cf4-g2wv6\" (UID: \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" Oct 31 13:26:50.970915 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.970904 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsfh5\" (UniqueName: \"kubernetes.io/projected/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-kube-api-access-jsfh5\") pod \"maas-api-revoke-sub-97538cf4-g2wv6\" (UID: \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" Oct 31 13:26:50.971439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.971416 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97538cf4-g2wv6\" (UID: \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" Oct 31 13:26:50.973086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.973065 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97538cf4-g2wv6\" (UID: \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" Oct 31 13:26:50.984228 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:50.984206 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsfh5\" (UniqueName: \"kubernetes.io/projected/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-kube-api-access-jsfh5\") pod \"maas-api-revoke-sub-97538cf4-g2wv6\" (UID: \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" Oct 31 13:26:51.071116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:51.071036 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" Oct 31 13:26:51.356009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:51.355983 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6"] Oct 31 13:26:51.357261 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:26:51.357237 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda197f417_2bd8_4495_b4cb_07e4cfbd0eb0.slice/crio-db630bb7467e1d4f483010ac0aa2651beaffc9d38e14af2171985c20cc9e29e8 WatchSource:0}: Error finding container db630bb7467e1d4f483010ac0aa2651beaffc9d38e14af2171985c20cc9e29e8: Status 404 returned error can't find the container with id db630bb7467e1d4f483010ac0aa2651beaffc9d38e14af2171985c20cc9e29e8 Oct 31 13:26:51.926159 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:51.926077 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" event={"ID":"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0","Type":"ContainerStarted","Data":"16b15ca4d55c41127e10c994ae81f968bd9559f6da1ac74d84cb9cc05b32170d"} Oct 31 13:26:51.926159 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:51.926128 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" event={"ID":"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0","Type":"ContainerStarted","Data":"db630bb7467e1d4f483010ac0aa2651beaffc9d38e14af2171985c20cc9e29e8"} Oct 31 13:26:51.945230 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:51.945175 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" podStartSLOduration=1.945159266 podStartE2EDuration="1.945159266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:51.945114459 +0000 UTC m=+1380.090858289" watchObservedRunningTime="2026-10-31 13:26:51.945159266 +0000 UTC m=+1380.090903094" Oct 31 13:26:52.933394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:52.933361 5532 generic.go:356] "Generic (PLEG): container finished" podID="a197f417-2bd8-4495-b4cb-07e4cfbd0eb0" containerID="16b15ca4d55c41127e10c994ae81f968bd9559f6da1ac74d84cb9cc05b32170d" exitCode=0 Oct 31 13:26:52.933394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:52.933398 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" event={"ID":"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0","Type":"ContainerDied","Data":"16b15ca4d55c41127e10c994ae81f968bd9559f6da1ac74d84cb9cc05b32170d"} Oct 31 13:26:54.129550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:54.129522 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" Oct 31 13:26:54.227966 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:54.227888 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-xks-ca\") pod \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\" (UID: \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\") " Oct 31 13:26:54.228100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:54.228064 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-openshift-ca\") pod \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\" (UID: \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\") " Oct 31 13:26:54.228169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:54.228155 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-kube-api-access-jsfh5\" (UniqueName: \"kubernetes.io/projected/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-kube-api-access-jsfh5\") pod \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\" (UID: \"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0\") " Oct 31 13:26:54.228409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:54.228384 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-openshift-ca" pod "a197f417-2bd8-4495-b4cb-07e4cfbd0eb0" (UID: "a197f417-2bd8-4495-b4cb-07e4cfbd0eb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:26:54.228549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:54.228533 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:54.230041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:54.230011 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-xks-ca" pod "a197f417-2bd8-4495-b4cb-07e4cfbd0eb0" (UID: "a197f417-2bd8-4495-b4cb-07e4cfbd0eb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:26:54.230133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:54.230061 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-kube-api-access-jsfh5" pod "a197f417-2bd8-4495-b4cb-07e4cfbd0eb0" (UID: "a197f417-2bd8-4495-b4cb-07e4cfbd0eb0"). InnerVolumeSpecName "kube-api-access-jsfh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:26:54.329728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:54.329692 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:54.329728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:54.329723 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsfh5\" (UniqueName: \"kubernetes.io/projected/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0-kube-api-access-jsfh5\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:26:54.950966 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:54.950934 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" Oct 31 13:26:54.950966 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:54.950952 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6" event={"ID":"a197f417-2bd8-4495-b4cb-07e4cfbd0eb0","Type":"ContainerDied","Data":"db630bb7467e1d4f483010ac0aa2651beaffc9d38e14af2171985c20cc9e29e8"} Oct 31 13:26:54.951185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:54.951004 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db630bb7467e1d4f483010ac0aa2651beaffc9d38e14af2171985c20cc9e29e8" Oct 31 13:26:56.882138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:56.882099 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22"] Oct 31 13:26:56.882855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:56.882833 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a197f417-2bd8-4495-b4cb-07e4cfbd0eb0" containerName="revoke-keys" Oct 31 13:26:56.882953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:56.882856 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a197f417-2bd8-4495-b4cb-07e4cfbd0eb0" containerName="revoke-keys" Oct 31 13:26:56.883051 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:56.883037 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a197f417-2bd8-4495-b4cb-07e4cfbd0eb0" containerName="revoke-keys" Oct 31 13:26:56.886494 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:56.886473 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" Oct 31 13:26:56.894424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:56.894404 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:26:56.894551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:56.894504 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:26:56.904998 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:56.904974 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22"] Oct 31 13:26:57.057215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.057181 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf350ff9-2tv22\" (UID: \"d9792277-7cb7-47e7-9e0a-0f14d934058d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" Oct 31 13:26:57.057355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.057325 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw6fb\" (UniqueName: \"kubernetes.io/projected/d9792277-7cb7-47e7-9e0a-0f14d934058d-kube-api-access-sw6fb\") pod \"maas-api-revoke-sub-bf350ff9-2tv22\" (UID: \"d9792277-7cb7-47e7-9e0a-0f14d934058d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" Oct 31 13:26:57.057439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.057421 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf350ff9-2tv22\" (UID: \"d9792277-7cb7-47e7-9e0a-0f14d934058d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" Oct 31 13:26:57.159034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.159002 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf350ff9-2tv22\" (UID: \"d9792277-7cb7-47e7-9e0a-0f14d934058d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" Oct 31 13:26:57.159207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.159067 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf350ff9-2tv22\" (UID: \"d9792277-7cb7-47e7-9e0a-0f14d934058d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" Oct 31 13:26:57.159207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.159160 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw6fb\" (UniqueName: \"kubernetes.io/projected/d9792277-7cb7-47e7-9e0a-0f14d934058d-kube-api-access-sw6fb\") pod \"maas-api-revoke-sub-bf350ff9-2tv22\" (UID: \"d9792277-7cb7-47e7-9e0a-0f14d934058d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" Oct 31 13:26:57.159808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.159783 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf350ff9-2tv22\" (UID: \"d9792277-7cb7-47e7-9e0a-0f14d934058d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" Oct 31 13:26:57.161383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.161359 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf350ff9-2tv22\" (UID: \"d9792277-7cb7-47e7-9e0a-0f14d934058d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" Oct 31 13:26:57.173243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.173215 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw6fb\" (UniqueName: \"kubernetes.io/projected/d9792277-7cb7-47e7-9e0a-0f14d934058d-kube-api-access-sw6fb\") pod \"maas-api-revoke-sub-bf350ff9-2tv22\" (UID: \"d9792277-7cb7-47e7-9e0a-0f14d934058d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" Oct 31 13:26:57.200086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.200063 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" Oct 31 13:26:57.483145 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.483116 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22"] Oct 31 13:26:57.486504 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:26:57.486394 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9792277_7cb7_47e7_9e0a_0f14d934058d.slice/crio-76756e2d1ee151c6fc7834cc7211824d61f09c3b63ce1db1783162cfbf4b616d WatchSource:0}: Error finding container 76756e2d1ee151c6fc7834cc7211824d61f09c3b63ce1db1783162cfbf4b616d: Status 404 returned error can't find the container with id 76756e2d1ee151c6fc7834cc7211824d61f09c3b63ce1db1783162cfbf4b616d Oct 31 13:26:57.541389 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.541364 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj"] Oct 31 13:26:57.548559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.548536 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" Oct 31 13:26:57.564123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.564100 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj"] Oct 31 13:26:57.665061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.665030 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvsrr\" (UniqueName: \"kubernetes.io/projected/34b79093-ec53-40ab-a284-c7d05a4c61f0-kube-api-access-lvsrr\") pod \"maas-api-revoke-sub-e0b86384-2p5bj\" (UID: \"34b79093-ec53-40ab-a284-c7d05a4c61f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" Oct 31 13:26:57.665231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.665073 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b86384-2p5bj\" (UID: \"34b79093-ec53-40ab-a284-c7d05a4c61f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" Oct 31 13:26:57.665311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.665289 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b86384-2p5bj\" (UID: \"34b79093-ec53-40ab-a284-c7d05a4c61f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" Oct 31 13:26:57.784617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.767193 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvsrr\" (UniqueName: \"kubernetes.io/projected/34b79093-ec53-40ab-a284-c7d05a4c61f0-kube-api-access-lvsrr\") pod \"maas-api-revoke-sub-e0b86384-2p5bj\" (UID: \"34b79093-ec53-40ab-a284-c7d05a4c61f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" Oct 31 13:26:57.784617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.767688 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b86384-2p5bj\" (UID: \"34b79093-ec53-40ab-a284-c7d05a4c61f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" Oct 31 13:26:57.784617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.770277 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b86384-2p5bj\" (UID: \"34b79093-ec53-40ab-a284-c7d05a4c61f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" Oct 31 13:26:57.784617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.771089 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b86384-2p5bj\" (UID: \"34b79093-ec53-40ab-a284-c7d05a4c61f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" Oct 31 13:26:57.784617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.772641 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b86384-2p5bj\" (UID: \"34b79093-ec53-40ab-a284-c7d05a4c61f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" Oct 31 13:26:57.784617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.781616 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvsrr\" (UniqueName: \"kubernetes.io/projected/34b79093-ec53-40ab-a284-c7d05a4c61f0-kube-api-access-lvsrr\") pod \"maas-api-revoke-sub-e0b86384-2p5bj\" (UID: \"34b79093-ec53-40ab-a284-c7d05a4c61f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" Oct 31 13:26:57.863889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.863865 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" Oct 31 13:26:57.970435 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.970398 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" event={"ID":"d9792277-7cb7-47e7-9e0a-0f14d934058d","Type":"ContainerStarted","Data":"13262a247314c8ee33f6ae6f789d1535df87a5fdf328cba42e1b7243c9689e79"} Oct 31 13:26:57.970855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:57.970466 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" event={"ID":"d9792277-7cb7-47e7-9e0a-0f14d934058d","Type":"ContainerStarted","Data":"76756e2d1ee151c6fc7834cc7211824d61f09c3b63ce1db1783162cfbf4b616d"} Oct 31 13:26:58.162936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:58.162870 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" podStartSLOduration=2.162851571 podStartE2EDuration="2.162851571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:58.007507393 +0000 UTC m=+1386.153251224" watchObservedRunningTime="2026-10-31 13:26:58.162851571 +0000 UTC m=+1386.308595401" Oct 31 13:26:58.165448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:58.165424 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj"] Oct 31 13:26:58.167796 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:26:58.167768 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34b79093_ec53_40ab_a284_c7d05a4c61f0.slice/crio-c34ed010740b8cc980cd24ff04379f792173fb3f3d6b95f340a26207a2123014 WatchSource:0}: Error finding container c34ed010740b8cc980cd24ff04379f792173fb3f3d6b95f340a26207a2123014: Status 404 returned error can't find the container with id c34ed010740b8cc980cd24ff04379f792173fb3f3d6b95f340a26207a2123014 Oct 31 13:26:58.977950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:58.977917 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" event={"ID":"34b79093-ec53-40ab-a284-c7d05a4c61f0","Type":"ContainerStarted","Data":"b7bc34469af3e2d6073870052ca332a6ae1a2dc3222e1dd840f3e94736ef491c"} Oct 31 13:26:58.978439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:58.977975 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" event={"ID":"34b79093-ec53-40ab-a284-c7d05a4c61f0","Type":"ContainerStarted","Data":"c34ed010740b8cc980cd24ff04379f792173fb3f3d6b95f340a26207a2123014"} Oct 31 13:26:58.979509 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:58.979476 5532 generic.go:356] "Generic (PLEG): container finished" podID="d9792277-7cb7-47e7-9e0a-0f14d934058d" containerID="13262a247314c8ee33f6ae6f789d1535df87a5fdf328cba42e1b7243c9689e79" exitCode=0 Oct 31 13:26:58.979613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:58.979522 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" event={"ID":"d9792277-7cb7-47e7-9e0a-0f14d934058d","Type":"ContainerDied","Data":"13262a247314c8ee33f6ae6f789d1535df87a5fdf328cba42e1b7243c9689e79"} Oct 31 13:26:58.998408 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:58.998362 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" podStartSLOduration=1.998345759 podStartE2EDuration="1.998345759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:26:58.996976159 +0000 UTC m=+1387.142719989" watchObservedRunningTime="2026-10-31 13:26:58.998345759 +0000 UTC m=+1387.144089587" Oct 31 13:26:59.707921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.707881 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4"] Oct 31 13:26:59.711879 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.711856 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" Oct 31 13:26:59.721637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.721609 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4"] Oct 31 13:26:59.799994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.799957 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-589d748d-jplz4\" (UID: \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" Oct 31 13:26:59.800177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.800036 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl7cz\" (UniqueName: \"kubernetes.io/projected/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-kube-api-access-sl7cz\") pod \"maas-api-revoke-sub-589d748d-jplz4\" (UID: \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" Oct 31 13:26:59.800271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.800244 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-589d748d-jplz4\" (UID: \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" Oct 31 13:26:59.900896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.900847 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl7cz\" (UniqueName: \"kubernetes.io/projected/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-kube-api-access-sl7cz\") pod \"maas-api-revoke-sub-589d748d-jplz4\" (UID: \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" Oct 31 13:26:59.901080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.900984 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-589d748d-jplz4\" (UID: \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" Oct 31 13:26:59.901080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.901027 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-589d748d-jplz4\" (UID: \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" Oct 31 13:26:59.901740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.901715 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-589d748d-jplz4\" (UID: \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" Oct 31 13:26:59.903237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.903213 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-589d748d-jplz4\" (UID: \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" Oct 31 13:26:59.917943 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.917917 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl7cz\" (UniqueName: \"kubernetes.io/projected/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-kube-api-access-sl7cz\") pod \"maas-api-revoke-sub-589d748d-jplz4\" (UID: \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" Oct 31 13:26:59.986783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.986696 5532 generic.go:356] "Generic (PLEG): container finished" podID="34b79093-ec53-40ab-a284-c7d05a4c61f0" containerID="b7bc34469af3e2d6073870052ca332a6ae1a2dc3222e1dd840f3e94736ef491c" exitCode=0 Oct 31 13:26:59.987147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:26:59.986789 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" event={"ID":"34b79093-ec53-40ab-a284-c7d05a4c61f0","Type":"ContainerDied","Data":"b7bc34469af3e2d6073870052ca332a6ae1a2dc3222e1dd840f3e94736ef491c"} Oct 31 13:27:00.026770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.026737 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" Oct 31 13:27:00.159267 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.159244 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" Oct 31 13:27:00.206984 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.206955 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-xks-ca\") pod \"d9792277-7cb7-47e7-9e0a-0f14d934058d\" (UID: \"d9792277-7cb7-47e7-9e0a-0f14d934058d\") " Oct 31 13:27:00.207125 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.207102 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-openshift-ca\") pod \"d9792277-7cb7-47e7-9e0a-0f14d934058d\" (UID: \"d9792277-7cb7-47e7-9e0a-0f14d934058d\") " Oct 31 13:27:00.207184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.207132 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9792277-7cb7-47e7-9e0a-0f14d934058d-kube-api-access-sw6fb\" (UniqueName: \"kubernetes.io/projected/d9792277-7cb7-47e7-9e0a-0f14d934058d-kube-api-access-sw6fb\") pod \"d9792277-7cb7-47e7-9e0a-0f14d934058d\" (UID: \"d9792277-7cb7-47e7-9e0a-0f14d934058d\") " Oct 31 13:27:00.207467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.207440 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-openshift-ca" pod "d9792277-7cb7-47e7-9e0a-0f14d934058d" (UID: "d9792277-7cb7-47e7-9e0a-0f14d934058d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:00.207571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.207559 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:00.209171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.209142 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-xks-ca" pod "d9792277-7cb7-47e7-9e0a-0f14d934058d" (UID: "d9792277-7cb7-47e7-9e0a-0f14d934058d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:27:00.209293 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.209272 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9792277-7cb7-47e7-9e0a-0f14d934058d-kube-api-access-sw6fb" pod "d9792277-7cb7-47e7-9e0a-0f14d934058d" (UID: "d9792277-7cb7-47e7-9e0a-0f14d934058d"). InnerVolumeSpecName "kube-api-access-sw6fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:00.308640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.308525 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9792277-7cb7-47e7-9e0a-0f14d934058d-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:00.308640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.308557 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw6fb\" (UniqueName: \"kubernetes.io/projected/d9792277-7cb7-47e7-9e0a-0f14d934058d-kube-api-access-sw6fb\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:00.333650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.333626 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4"] Oct 31 13:27:00.335698 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:00.335660 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1cc949a_c1bb_41e7_8a85_465ff45fbfb2.slice/crio-b32f4ecde182ffb213ae51f2ce230975496df0f8924c8c52c9888b2b36ed765a WatchSource:0}: Error finding container b32f4ecde182ffb213ae51f2ce230975496df0f8924c8c52c9888b2b36ed765a: Status 404 returned error can't find the container with id b32f4ecde182ffb213ae51f2ce230975496df0f8924c8c52c9888b2b36ed765a Oct 31 13:27:00.994439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.994345 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" event={"ID":"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2","Type":"ContainerStarted","Data":"4e24ed62464b25ff868db7295360943e051b3ba651fb4920f730a33d30d7cfcc"} Oct 31 13:27:00.994439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.994407 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" event={"ID":"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2","Type":"ContainerStarted","Data":"b32f4ecde182ffb213ae51f2ce230975496df0f8924c8c52c9888b2b36ed765a"} Oct 31 13:27:00.995803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.995773 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" event={"ID":"d9792277-7cb7-47e7-9e0a-0f14d934058d","Type":"ContainerDied","Data":"76756e2d1ee151c6fc7834cc7211824d61f09c3b63ce1db1783162cfbf4b616d"} Oct 31 13:27:00.995803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.995800 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22" Oct 31 13:27:00.996008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:00.995812 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="76756e2d1ee151c6fc7834cc7211824d61f09c3b63ce1db1783162cfbf4b616d" Oct 31 13:27:01.017141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:01.017074 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" podStartSLOduration=2.017055442 podStartE2EDuration="2.017055442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:26:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:01.016322595 +0000 UTC m=+1389.162066423" watchObservedRunningTime="2026-10-31 13:27:01.017055442 +0000 UTC m=+1389.162799271" Oct 31 13:27:01.130352 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:01.130310 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" Oct 31 13:27:01.220028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:01.219994 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-xks-ca\") pod \"34b79093-ec53-40ab-a284-c7d05a4c61f0\" (UID: \"34b79093-ec53-40ab-a284-c7d05a4c61f0\") " Oct 31 13:27:01.220198 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:01.220076 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-openshift-ca\") pod \"34b79093-ec53-40ab-a284-c7d05a4c61f0\" (UID: \"34b79093-ec53-40ab-a284-c7d05a4c61f0\") " Oct 31 13:27:01.220268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:01.220227 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34b79093-ec53-40ab-a284-c7d05a4c61f0-kube-api-access-lvsrr\" (UniqueName: \"kubernetes.io/projected/34b79093-ec53-40ab-a284-c7d05a4c61f0-kube-api-access-lvsrr\") pod \"34b79093-ec53-40ab-a284-c7d05a4c61f0\" (UID: \"34b79093-ec53-40ab-a284-c7d05a4c61f0\") " Oct 31 13:27:01.220425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:01.220399 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-openshift-ca" pod "34b79093-ec53-40ab-a284-c7d05a4c61f0" (UID: "34b79093-ec53-40ab-a284-c7d05a4c61f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:01.220668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:01.220651 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:01.222096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:01.222071 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-xks-ca" pod "34b79093-ec53-40ab-a284-c7d05a4c61f0" (UID: "34b79093-ec53-40ab-a284-c7d05a4c61f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:27:01.222178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:01.222135 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34b79093-ec53-40ab-a284-c7d05a4c61f0-kube-api-access-lvsrr" pod "34b79093-ec53-40ab-a284-c7d05a4c61f0" (UID: "34b79093-ec53-40ab-a284-c7d05a4c61f0"). InnerVolumeSpecName "kube-api-access-lvsrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:01.322315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:01.322283 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34b79093-ec53-40ab-a284-c7d05a4c61f0-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:01.322315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:01.322312 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvsrr\" (UniqueName: \"kubernetes.io/projected/34b79093-ec53-40ab-a284-c7d05a4c61f0-kube-api-access-lvsrr\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:01.761573 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:27:01.761538 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:27:02.002842 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:02.002807 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" Oct 31 13:27:02.003283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:02.003220 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj" event={"ID":"34b79093-ec53-40ab-a284-c7d05a4c61f0","Type":"ContainerDied","Data":"c34ed010740b8cc980cd24ff04379f792173fb3f3d6b95f340a26207a2123014"} Oct 31 13:27:02.003283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:02.003265 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c34ed010740b8cc980cd24ff04379f792173fb3f3d6b95f340a26207a2123014" Oct 31 13:27:02.004436 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:02.004408 5532 generic.go:356] "Generic (PLEG): container finished" podID="b1cc949a-c1bb-41e7-8a85-465ff45fbfb2" containerID="4e24ed62464b25ff868db7295360943e051b3ba651fb4920f730a33d30d7cfcc" exitCode=0 Oct 31 13:27:02.004632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:02.004477 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" event={"ID":"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2","Type":"ContainerDied","Data":"4e24ed62464b25ff868db7295360943e051b3ba651fb4920f730a33d30d7cfcc"} Oct 31 13:27:03.147941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:03.147917 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" Oct 31 13:27:03.243767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:03.243733 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-openshift-ca\") pod \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\" (UID: \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\") " Oct 31 13:27:03.243953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:03.243791 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-kube-api-access-sl7cz\" (UniqueName: \"kubernetes.io/projected/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-kube-api-access-sl7cz\") pod \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\" (UID: \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\") " Oct 31 13:27:03.243953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:03.243831 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-xks-ca\") pod \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\" (UID: \"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2\") " Oct 31 13:27:03.244160 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:03.244136 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-openshift-ca" pod "b1cc949a-c1bb-41e7-8a85-465ff45fbfb2" (UID: "b1cc949a-c1bb-41e7-8a85-465ff45fbfb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:03.244304 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:03.244287 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:03.246030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:03.246011 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-kube-api-access-sl7cz" pod "b1cc949a-c1bb-41e7-8a85-465ff45fbfb2" (UID: "b1cc949a-c1bb-41e7-8a85-465ff45fbfb2"). InnerVolumeSpecName "kube-api-access-sl7cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:03.246151 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:03.246120 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-xks-ca" pod "b1cc949a-c1bb-41e7-8a85-465ff45fbfb2" (UID: "b1cc949a-c1bb-41e7-8a85-465ff45fbfb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:27:03.345245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:03.345166 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl7cz\" (UniqueName: \"kubernetes.io/projected/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-kube-api-access-sl7cz\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:03.345245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:03.345192 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:04.017094 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:04.017065 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" event={"ID":"b1cc949a-c1bb-41e7-8a85-465ff45fbfb2","Type":"ContainerDied","Data":"b32f4ecde182ffb213ae51f2ce230975496df0f8924c8c52c9888b2b36ed765a"} Oct 31 13:27:04.017210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:04.017100 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4" Oct 31 13:27:04.017210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:04.017111 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b32f4ecde182ffb213ae51f2ce230975496df0f8924c8c52c9888b2b36ed765a" Oct 31 13:27:05.226694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.226654 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x"] Oct 31 13:27:05.227361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.227340 5532 cpu_manager.go:414] "containerMap: removing container" podUID="34b79093-ec53-40ab-a284-c7d05a4c61f0" containerName="revoke-keys" Oct 31 13:27:05.227470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.227362 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34b79093-ec53-40ab-a284-c7d05a4c61f0" containerName="revoke-keys" Oct 31 13:27:05.227470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.227387 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d9792277-7cb7-47e7-9e0a-0f14d934058d" containerName="revoke-keys" Oct 31 13:27:05.227470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.227395 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9792277-7cb7-47e7-9e0a-0f14d934058d" containerName="revoke-keys" Oct 31 13:27:05.227470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.227406 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b1cc949a-c1bb-41e7-8a85-465ff45fbfb2" containerName="revoke-keys" Oct 31 13:27:05.227470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.227413 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1cc949a-c1bb-41e7-8a85-465ff45fbfb2" containerName="revoke-keys" Oct 31 13:27:05.227766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.227571 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1cc949a-c1bb-41e7-8a85-465ff45fbfb2" containerName="revoke-keys" Oct 31 13:27:05.227766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.227589 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9792277-7cb7-47e7-9e0a-0f14d934058d" containerName="revoke-keys" Oct 31 13:27:05.227766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.227635 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="34b79093-ec53-40ab-a284-c7d05a4c61f0" containerName="revoke-keys" Oct 31 13:27:05.231074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.231053 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" Oct 31 13:27:05.239633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.239571 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:27:05.240638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.240590 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:27:05.243228 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.243206 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x"] Oct 31 13:27:05.267543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.267505 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw78z\" (UniqueName: \"kubernetes.io/projected/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-kube-api-access-bw78z\") pod \"maas-api-revoke-sub-d4e7a530-6vp6x\" (UID: \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" Oct 31 13:27:05.267734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.267713 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e7a530-6vp6x\" (UID: \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" Oct 31 13:27:05.267888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.267871 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e7a530-6vp6x\" (UID: \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" Oct 31 13:27:05.368546 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.368512 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e7a530-6vp6x\" (UID: \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" Oct 31 13:27:05.368771 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.368574 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw78z\" (UniqueName: \"kubernetes.io/projected/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-kube-api-access-bw78z\") pod \"maas-api-revoke-sub-d4e7a530-6vp6x\" (UID: \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" Oct 31 13:27:05.368843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.368791 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e7a530-6vp6x\" (UID: \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" Oct 31 13:27:05.369339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.369316 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e7a530-6vp6x\" (UID: \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" Oct 31 13:27:05.371251 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.371222 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e7a530-6vp6x\" (UID: \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" Oct 31 13:27:05.384137 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.384114 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw78z\" (UniqueName: \"kubernetes.io/projected/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-kube-api-access-bw78z\") pod \"maas-api-revoke-sub-d4e7a530-6vp6x\" (UID: \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" Oct 31 13:27:05.547210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.547125 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" Oct 31 13:27:05.826810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:05.826782 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x"] Oct 31 13:27:05.828475 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:05.828450 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc83ddeaf_165a_4f00_9a9a_2ac8bce136ca.slice/crio-3d836166e48614db8f31051c82b15054beab877733ecb0ca528a3ee76d284484 WatchSource:0}: Error finding container 3d836166e48614db8f31051c82b15054beab877733ecb0ca528a3ee76d284484: Status 404 returned error can't find the container with id 3d836166e48614db8f31051c82b15054beab877733ecb0ca528a3ee76d284484 Oct 31 13:27:06.029726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:06.029687 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" event={"ID":"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca","Type":"ContainerStarted","Data":"3d836166e48614db8f31051c82b15054beab877733ecb0ca528a3ee76d284484"} Oct 31 13:27:07.038287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:07.038252 5532 generic.go:356] "Generic (PLEG): container finished" podID="c83ddeaf-165a-4f00-9a9a-2ac8bce136ca" containerID="906bb7b5e191ea65ce5731aff8a60ac18155789b25ba1f7dd5f1147cef80177e" exitCode=0 Oct 31 13:27:07.038663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:07.038328 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" event={"ID":"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca","Type":"ContainerDied","Data":"906bb7b5e191ea65ce5731aff8a60ac18155789b25ba1f7dd5f1147cef80177e"} Oct 31 13:27:08.191165 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:08.191139 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" Oct 31 13:27:08.300669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:08.300567 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-xks-ca\") pod \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\" (UID: \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\") " Oct 31 13:27:08.300806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:08.300722 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-openshift-ca\") pod \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\" (UID: \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\") " Oct 31 13:27:08.300853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:08.300831 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-kube-api-access-bw78z\" (UniqueName: \"kubernetes.io/projected/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-kube-api-access-bw78z\") pod \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\" (UID: \"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca\") " Oct 31 13:27:08.301100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:08.301069 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-openshift-ca" pod "c83ddeaf-165a-4f00-9a9a-2ac8bce136ca" (UID: "c83ddeaf-165a-4f00-9a9a-2ac8bce136ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:08.301272 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:08.301256 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:08.302577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:08.302555 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-xks-ca" pod "c83ddeaf-165a-4f00-9a9a-2ac8bce136ca" (UID: "c83ddeaf-165a-4f00-9a9a-2ac8bce136ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:27:08.302827 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:08.302811 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-kube-api-access-bw78z" pod "c83ddeaf-165a-4f00-9a9a-2ac8bce136ca" (UID: "c83ddeaf-165a-4f00-9a9a-2ac8bce136ca"). InnerVolumeSpecName "kube-api-access-bw78z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:08.401988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:08.401958 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw78z\" (UniqueName: \"kubernetes.io/projected/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-kube-api-access-bw78z\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:08.401988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:08.401987 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:09.051665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:09.051629 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" Oct 31 13:27:09.051852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:09.051632 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x" event={"ID":"c83ddeaf-165a-4f00-9a9a-2ac8bce136ca","Type":"ContainerDied","Data":"3d836166e48614db8f31051c82b15054beab877733ecb0ca528a3ee76d284484"} Oct 31 13:27:09.051852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:09.051766 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d836166e48614db8f31051c82b15054beab877733ecb0ca528a3ee76d284484" Oct 31 13:27:10.015102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.015069 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:27:10.038446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.038410 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr"] Oct 31 13:27:10.038917 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.038902 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c83ddeaf-165a-4f00-9a9a-2ac8bce136ca" containerName="revoke-keys" Oct 31 13:27:10.038979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.038918 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83ddeaf-165a-4f00-9a9a-2ac8bce136ca" containerName="revoke-keys" Oct 31 13:27:10.039059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.039050 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83ddeaf-165a-4f00-9a9a-2ac8bce136ca" containerName="revoke-keys" Oct 31 13:27:10.046984 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.046964 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" Oct 31 13:27:10.053736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.053709 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr"] Oct 31 13:27:10.054007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.053991 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:27:10.054257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.054239 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:27:10.121540 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.121500 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4fb8bcb-r8kdr\" (UID: \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" Oct 31 13:27:10.121797 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.121583 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4fb8bcb-r8kdr\" (UID: \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" Oct 31 13:27:10.121884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.121855 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nlm6\" (UniqueName: \"kubernetes.io/projected/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-kube-api-access-9nlm6\") pod \"maas-api-revoke-sub-b4fb8bcb-r8kdr\" (UID: \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" Oct 31 13:27:10.223398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.223362 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nlm6\" (UniqueName: \"kubernetes.io/projected/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-kube-api-access-9nlm6\") pod \"maas-api-revoke-sub-b4fb8bcb-r8kdr\" (UID: \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" Oct 31 13:27:10.223585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.223469 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4fb8bcb-r8kdr\" (UID: \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" Oct 31 13:27:10.223585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.223513 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4fb8bcb-r8kdr\" (UID: \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" Oct 31 13:27:10.224160 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.224138 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b4fb8bcb-r8kdr\" (UID: \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" Oct 31 13:27:10.225787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.225766 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b4fb8bcb-r8kdr\" (UID: \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" Oct 31 13:27:10.237271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.237250 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nlm6\" (UniqueName: \"kubernetes.io/projected/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-kube-api-access-9nlm6\") pod \"maas-api-revoke-sub-b4fb8bcb-r8kdr\" (UID: \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" Oct 31 13:27:10.366478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.366399 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" Oct 31 13:27:10.656854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:10.656827 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr"] Oct 31 13:27:10.659472 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:10.659437 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9a5c2f_4082_421d_99b2_3169fd3b4c07.slice/crio-39fb93b963065ab17d2af7dd2cebd1ee9060a7ccf7f5a1663b26fe9a6db845b4 WatchSource:0}: Error finding container 39fb93b963065ab17d2af7dd2cebd1ee9060a7ccf7f5a1663b26fe9a6db845b4: Status 404 returned error can't find the container with id 39fb93b963065ab17d2af7dd2cebd1ee9060a7ccf7f5a1663b26fe9a6db845b4 Oct 31 13:27:11.105615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:11.105179 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" event={"ID":"6d9a5c2f-4082-421d-99b2-3169fd3b4c07","Type":"ContainerStarted","Data":"399039575ab60a47e385a416438be0150834fcaa3f66e2b19d14fa2bf934937b"} Oct 31 13:27:11.105615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:11.105249 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" event={"ID":"6d9a5c2f-4082-421d-99b2-3169fd3b4c07","Type":"ContainerStarted","Data":"39fb93b963065ab17d2af7dd2cebd1ee9060a7ccf7f5a1663b26fe9a6db845b4"} Oct 31 13:27:11.134611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:11.134540 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" podStartSLOduration=1.1345210510000001 podStartE2EDuration="1.134521051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:11.132892609 +0000 UTC m=+1399.278636439" watchObservedRunningTime="2026-10-31 13:27:11.134521051 +0000 UTC m=+1399.280264877" Oct 31 13:27:12.124787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:12.124668 5532 generic.go:356] "Generic (PLEG): container finished" podID="6d9a5c2f-4082-421d-99b2-3169fd3b4c07" containerID="399039575ab60a47e385a416438be0150834fcaa3f66e2b19d14fa2bf934937b" exitCode=0 Oct 31 13:27:12.125617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:12.125501 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" event={"ID":"6d9a5c2f-4082-421d-99b2-3169fd3b4c07","Type":"ContainerDied","Data":"399039575ab60a47e385a416438be0150834fcaa3f66e2b19d14fa2bf934937b"} Oct 31 13:27:13.469313 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.469278 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5"] Oct 31 13:27:13.507664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.502832 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.513358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.513327 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c82140-openshift-default-dockercfg-bmc79\"" Oct 31 13:27:13.632617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.604306 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5"] Oct 31 13:27:13.657620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.631981 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.657961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.657929 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.682713 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.682674 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42c8c\" (UniqueName: \"kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-kube-api-access-42c8c\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.686616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.683019 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.686800 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.686767 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40c874b-6a3b-41db-8415-7db99e7dcf71-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.697625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.687032 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.697911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.697879 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.745734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.701670 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.746330 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.746299 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.850463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.848762 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.850463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.848810 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42c8c\" (UniqueName: \"kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-kube-api-access-42c8c\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.850463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.848904 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.850463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.848967 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40c874b-6a3b-41db-8415-7db99e7dcf71-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.850463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.849035 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.850463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.849085 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.850463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.849110 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.850463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.849137 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.850463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.849191 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.850463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.849669 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.852932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.852055 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.854486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.853481 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40c874b-6a3b-41db-8415-7db99e7dcf71-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.854486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.854167 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.854486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.854432 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.875757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.875675 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42c8c\" (UniqueName: \"kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-kube-api-access-42c8c\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.895617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.880607 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.935650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.929218 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:13.935650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:13.929777 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:14.176619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.171766 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:14.226700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.226668 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" Oct 31 13:27:14.300238 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.299168 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-openshift-ca\") pod \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\" (UID: \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\") " Oct 31 13:27:14.300238 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.299284 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-kube-api-access-9nlm6\" (UniqueName: \"kubernetes.io/projected/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-kube-api-access-9nlm6\") pod \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\" (UID: \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\") " Oct 31 13:27:14.300238 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.299416 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-xks-ca\") pod \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\" (UID: \"6d9a5c2f-4082-421d-99b2-3169fd3b4c07\") " Oct 31 13:27:14.321609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.321542 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-openshift-ca" pod "6d9a5c2f-4082-421d-99b2-3169fd3b4c07" (UID: "6d9a5c2f-4082-421d-99b2-3169fd3b4c07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:14.356613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.350470 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-xks-ca" pod "6d9a5c2f-4082-421d-99b2-3169fd3b4c07" (UID: "6d9a5c2f-4082-421d-99b2-3169fd3b4c07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:27:14.356613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.350721 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-kube-api-access-9nlm6" pod "6d9a5c2f-4082-421d-99b2-3169fd3b4c07" (UID: "6d9a5c2f-4082-421d-99b2-3169fd3b4c07"). InnerVolumeSpecName "kube-api-access-9nlm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:14.389067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.388966 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" event={"ID":"6d9a5c2f-4082-421d-99b2-3169fd3b4c07","Type":"ContainerDied","Data":"39fb93b963065ab17d2af7dd2cebd1ee9060a7ccf7f5a1663b26fe9a6db845b4"} Oct 31 13:27:14.389337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.389302 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39fb93b963065ab17d2af7dd2cebd1ee9060a7ccf7f5a1663b26fe9a6db845b4" Oct 31 13:27:14.389554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.389537 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr" Oct 31 13:27:14.430205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.419305 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:14.430205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.419339 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nlm6\" (UniqueName: \"kubernetes.io/projected/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-kube-api-access-9nlm6\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:14.430205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.419355 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d9a5c2f-4082-421d-99b2-3169fd3b4c07-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:14.698957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.698927 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5"] Oct 31 13:27:14.700490 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:14.700463 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode40c874b_6a3b_41db_8415_7db99e7dcf71.slice/crio-0628a8e41a61180b0b91f6054eb30d25c181ad7a9d5f75cdca7fa18c925d3b32 WatchSource:0}: Error finding container 0628a8e41a61180b0b91f6054eb30d25c181ad7a9d5f75cdca7fa18c925d3b32: Status 404 returned error can't find the container with id 0628a8e41a61180b0b91f6054eb30d25c181ad7a9d5f75cdca7fa18c925d3b32 Oct 31 13:27:14.702979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.702952 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:14.703037 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.703009 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:14.703069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:14.703040 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:15.398290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.398243 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" event={"ID":"e40c874b-6a3b-41db-8415-7db99e7dcf71","Type":"ContainerStarted","Data":"21906d21329a411f77572048159e5a3c7e43a292a620698652aa28189a990bc3"} Oct 31 13:27:15.398457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.398306 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" event={"ID":"e40c874b-6a3b-41db-8415-7db99e7dcf71","Type":"ContainerStarted","Data":"0628a8e41a61180b0b91f6054eb30d25c181ad7a9d5f75cdca7fa18c925d3b32"} Oct 31 13:27:15.425947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.425884 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" podStartSLOduration=2.425865998 podStartE2EDuration="2.425865998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:15.42497545 +0000 UTC m=+1403.570719280" watchObservedRunningTime="2026-10-31 13:27:15.425865998 +0000 UTC m=+1403.571609828" Oct 31 13:27:15.479232 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.479193 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6"] Oct 31 13:27:15.479817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.479801 5532 cpu_manager.go:414] "containerMap: removing container" podUID="6d9a5c2f-4082-421d-99b2-3169fd3b4c07" containerName="revoke-keys" Oct 31 13:27:15.479862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.479818 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d9a5c2f-4082-421d-99b2-3169fd3b4c07" containerName="revoke-keys" Oct 31 13:27:15.479951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.479938 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d9a5c2f-4082-421d-99b2-3169fd3b4c07" containerName="revoke-keys" Oct 31 13:27:15.483516 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.483500 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" Oct 31 13:27:15.490133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.490081 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:27:15.490284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.490228 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:27:15.493781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.493755 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6"] Oct 31 13:27:15.534029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.533995 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22dbacb2-h68k6\" (UID: \"d3c258c3-2446-4fab-aa37-50b687f9651b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" Oct 31 13:27:15.534181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.534051 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22dbacb2-h68k6\" (UID: \"d3c258c3-2446-4fab-aa37-50b687f9651b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" Oct 31 13:27:15.534181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.534118 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtnzl\" (UniqueName: \"kubernetes.io/projected/d3c258c3-2446-4fab-aa37-50b687f9651b-kube-api-access-vtnzl\") pod \"maas-api-revoke-sub-22dbacb2-h68k6\" (UID: \"d3c258c3-2446-4fab-aa37-50b687f9651b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" Oct 31 13:27:15.634927 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.634891 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22dbacb2-h68k6\" (UID: \"d3c258c3-2446-4fab-aa37-50b687f9651b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" Oct 31 13:27:15.635092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.634949 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22dbacb2-h68k6\" (UID: \"d3c258c3-2446-4fab-aa37-50b687f9651b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" Oct 31 13:27:15.635092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.634965 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtnzl\" (UniqueName: \"kubernetes.io/projected/d3c258c3-2446-4fab-aa37-50b687f9651b-kube-api-access-vtnzl\") pod \"maas-api-revoke-sub-22dbacb2-h68k6\" (UID: \"d3c258c3-2446-4fab-aa37-50b687f9651b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" Oct 31 13:27:15.635718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.635697 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22dbacb2-h68k6\" (UID: \"d3c258c3-2446-4fab-aa37-50b687f9651b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" Oct 31 13:27:15.637443 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.637426 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22dbacb2-h68k6\" (UID: \"d3c258c3-2446-4fab-aa37-50b687f9651b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" Oct 31 13:27:15.666512 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.666427 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtnzl\" (UniqueName: \"kubernetes.io/projected/d3c258c3-2446-4fab-aa37-50b687f9651b-kube-api-access-vtnzl\") pod \"maas-api-revoke-sub-22dbacb2-h68k6\" (UID: \"d3c258c3-2446-4fab-aa37-50b687f9651b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" Oct 31 13:27:15.800125 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:15.800083 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" Oct 31 13:27:16.104099 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:16.104070 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6"] Oct 31 13:27:16.104744 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:16.104720 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c258c3_2446_4fab_aa37_50b687f9651b.slice/crio-cdbb7ddb71fce0e0431b43fd7c17249bedc7607eb15c3b3795292939c2f6b149 WatchSource:0}: Error finding container cdbb7ddb71fce0e0431b43fd7c17249bedc7607eb15c3b3795292939c2f6b149: Status 404 returned error can't find the container with id cdbb7ddb71fce0e0431b43fd7c17249bedc7607eb15c3b3795292939c2f6b149 Oct 31 13:27:16.172865 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:16.172831 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:16.174536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:16.174508 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:27:16.174652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:16.174561 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" podUID="e40c874b-6a3b-41db-8415-7db99e7dcf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 13:27:16.405635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:16.405574 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" event={"ID":"d3c258c3-2446-4fab-aa37-50b687f9651b","Type":"ContainerStarted","Data":"cdbb7ddb71fce0e0431b43fd7c17249bedc7607eb15c3b3795292939c2f6b149"} Oct 31 13:27:17.173005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:17.172974 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:27:17.173553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:17.173517 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" podUID="e40c874b-6a3b-41db-8415-7db99e7dcf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 13:27:17.414835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:17.414689 5532 generic.go:356] "Generic (PLEG): container finished" podID="d3c258c3-2446-4fab-aa37-50b687f9651b" containerID="e25864b0c149da5f20659da4046e28fde573bb4eb2bd50232bc8c633fcd10a91" exitCode=0 Oct 31 13:27:17.414835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:17.414768 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" event={"ID":"d3c258c3-2446-4fab-aa37-50b687f9651b","Type":"ContainerDied","Data":"e25864b0c149da5f20659da4046e28fde573bb4eb2bd50232bc8c633fcd10a91"} Oct 31 13:27:18.172774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:18.172738 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:27:18.172932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:18.172788 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" podUID="e40c874b-6a3b-41db-8415-7db99e7dcf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 13:27:18.615373 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:18.615346 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" Oct 31 13:27:18.697237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:18.697203 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-xks-ca\") pod \"d3c258c3-2446-4fab-aa37-50b687f9651b\" (UID: \"d3c258c3-2446-4fab-aa37-50b687f9651b\") " Oct 31 13:27:18.697465 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:18.697367 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c258c3-2446-4fab-aa37-50b687f9651b-kube-api-access-vtnzl\" (UniqueName: \"kubernetes.io/projected/d3c258c3-2446-4fab-aa37-50b687f9651b-kube-api-access-vtnzl\") pod \"d3c258c3-2446-4fab-aa37-50b687f9651b\" (UID: \"d3c258c3-2446-4fab-aa37-50b687f9651b\") " Oct 31 13:27:18.697465 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:18.697397 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-openshift-ca\") pod \"d3c258c3-2446-4fab-aa37-50b687f9651b\" (UID: \"d3c258c3-2446-4fab-aa37-50b687f9651b\") " Oct 31 13:27:18.697879 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:18.697840 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-openshift-ca" pod "d3c258c3-2446-4fab-aa37-50b687f9651b" (UID: "d3c258c3-2446-4fab-aa37-50b687f9651b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:18.699266 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:18.699242 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-xks-ca" pod "d3c258c3-2446-4fab-aa37-50b687f9651b" (UID: "d3c258c3-2446-4fab-aa37-50b687f9651b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:27:18.699401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:18.699384 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c258c3-2446-4fab-aa37-50b687f9651b-kube-api-access-vtnzl" pod "d3c258c3-2446-4fab-aa37-50b687f9651b" (UID: "d3c258c3-2446-4fab-aa37-50b687f9651b"). InnerVolumeSpecName "kube-api-access-vtnzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:18.799459 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:18.799384 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtnzl\" (UniqueName: \"kubernetes.io/projected/d3c258c3-2446-4fab-aa37-50b687f9651b-kube-api-access-vtnzl\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:18.799459 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:18.799414 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:18.799459 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:18.799425 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3c258c3-2446-4fab-aa37-50b687f9651b-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:19.173033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:19.173004 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:27:19.173206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:19.173052 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" podUID="e40c874b-6a3b-41db-8415-7db99e7dcf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 13:27:19.428090 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:19.428006 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" Oct 31 13:27:19.428090 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:19.428027 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6" event={"ID":"d3c258c3-2446-4fab-aa37-50b687f9651b","Type":"ContainerDied","Data":"cdbb7ddb71fce0e0431b43fd7c17249bedc7607eb15c3b3795292939c2f6b149"} Oct 31 13:27:19.428279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:19.428099 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdbb7ddb71fce0e0431b43fd7c17249bedc7607eb15c3b3795292939c2f6b149" Oct 31 13:27:19.880951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:19.880877 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8"] Oct 31 13:27:19.881488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:19.881472 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d3c258c3-2446-4fab-aa37-50b687f9651b" containerName="revoke-keys" Oct 31 13:27:19.881534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:19.881489 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c258c3-2446-4fab-aa37-50b687f9651b" containerName="revoke-keys" Oct 31 13:27:19.881574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:19.881565 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c258c3-2446-4fab-aa37-50b687f9651b" containerName="revoke-keys" Oct 31 13:27:19.885127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:19.885112 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" Oct 31 13:27:19.891265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:19.891238 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:27:19.891407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:19.891384 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:27:19.895254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:19.895227 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8"] Oct 31 13:27:20.012046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:20.012016 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkcdn\" (UniqueName: \"kubernetes.io/projected/41cb130e-a0d3-44dc-9e50-bc571de0c55f-kube-api-access-mkcdn\") pod \"maas-api-revoke-sub-8f107cfb-zbrz8\" (UID: \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" Oct 31 13:27:20.012232 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:20.012162 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f107cfb-zbrz8\" (UID: \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" Oct 31 13:27:20.012323 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:20.012301 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f107cfb-zbrz8\" (UID: \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" Oct 31 13:27:20.113634 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:20.113576 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f107cfb-zbrz8\" (UID: \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" Oct 31 13:27:20.113839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:20.113700 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkcdn\" (UniqueName: \"kubernetes.io/projected/41cb130e-a0d3-44dc-9e50-bc571de0c55f-kube-api-access-mkcdn\") pod \"maas-api-revoke-sub-8f107cfb-zbrz8\" (UID: \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" Oct 31 13:27:20.113839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:20.113794 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f107cfb-zbrz8\" (UID: \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" Oct 31 13:27:20.114640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:20.114613 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f107cfb-zbrz8\" (UID: \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" Oct 31 13:27:20.116099 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:20.116074 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f107cfb-zbrz8\" (UID: \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" Oct 31 13:27:20.129110 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:20.129085 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkcdn\" (UniqueName: \"kubernetes.io/projected/41cb130e-a0d3-44dc-9e50-bc571de0c55f-kube-api-access-mkcdn\") pod \"maas-api-revoke-sub-8f107cfb-zbrz8\" (UID: \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" Oct 31 13:27:20.172793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:20.172755 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:27:20.172976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:20.172810 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" podUID="e40c874b-6a3b-41db-8415-7db99e7dcf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 13:27:20.200608 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:20.200557 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" Oct 31 13:27:20.519803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:20.519779 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8"] Oct 31 13:27:20.522477 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:20.522451 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cb130e_a0d3_44dc_9e50_bc571de0c55f.slice/crio-9e8135a5718dee06f0fa75d261a0b39043f99b692b6741f68f8c34b5db728344 WatchSource:0}: Error finding container 9e8135a5718dee06f0fa75d261a0b39043f99b692b6741f68f8c34b5db728344: Status 404 returned error can't find the container with id 9e8135a5718dee06f0fa75d261a0b39043f99b692b6741f68f8c34b5db728344 Oct 31 13:27:21.172992 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:21.172955 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body= Oct 31 13:27:21.173376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:21.173009 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" podUID="e40c874b-6a3b-41db-8415-7db99e7dcf71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" Oct 31 13:27:21.443953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:21.443913 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" event={"ID":"41cb130e-a0d3-44dc-9e50-bc571de0c55f","Type":"ContainerStarted","Data":"082c00227f8d21cf24acbf4eecc792efdeeaf73328fa42a62c8eccd177bf5e13"} Oct 31 13:27:21.444154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:21.443984 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" event={"ID":"41cb130e-a0d3-44dc-9e50-bc571de0c55f","Type":"ContainerStarted","Data":"9e8135a5718dee06f0fa75d261a0b39043f99b692b6741f68f8c34b5db728344"} Oct 31 13:27:21.467672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:21.467573 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" podStartSLOduration=2.467556472 podStartE2EDuration="2.467556472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:21.463153764 +0000 UTC m=+1409.608897603" watchObservedRunningTime="2026-10-31 13:27:21.467556472 +0000 UTC m=+1409.613300301" Oct 31 13:27:22.176888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:22.176860 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:22.177349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:22.177155 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:22.177977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:22.177957 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 13:27:22.451835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:22.451749 5532 generic.go:356] "Generic (PLEG): container finished" podID="41cb130e-a0d3-44dc-9e50-bc571de0c55f" containerID="082c00227f8d21cf24acbf4eecc792efdeeaf73328fa42a62c8eccd177bf5e13" exitCode=0 Oct 31 13:27:22.451998 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:22.451880 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" event={"ID":"41cb130e-a0d3-44dc-9e50-bc571de0c55f","Type":"ContainerDied","Data":"082c00227f8d21cf24acbf4eecc792efdeeaf73328fa42a62c8eccd177bf5e13"} Oct 31 13:27:23.618420 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:23.618396 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" Oct 31 13:27:23.763545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:23.763449 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-xks-ca\") pod \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\" (UID: \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\") " Oct 31 13:27:23.763754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:23.763629 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-openshift-ca\") pod \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\" (UID: \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\") " Oct 31 13:27:23.763825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:23.763783 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cb130e-a0d3-44dc-9e50-bc571de0c55f-kube-api-access-mkcdn\" (UniqueName: \"kubernetes.io/projected/41cb130e-a0d3-44dc-9e50-bc571de0c55f-kube-api-access-mkcdn\") pod \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\" (UID: \"41cb130e-a0d3-44dc-9e50-bc571de0c55f\") " Oct 31 13:27:23.763977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:23.763951 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-openshift-ca" pod "41cb130e-a0d3-44dc-9e50-bc571de0c55f" (UID: "41cb130e-a0d3-44dc-9e50-bc571de0c55f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:23.764232 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:23.764214 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:23.765552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:23.765527 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-xks-ca" pod "41cb130e-a0d3-44dc-9e50-bc571de0c55f" (UID: "41cb130e-a0d3-44dc-9e50-bc571de0c55f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:27:23.765798 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:23.765780 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cb130e-a0d3-44dc-9e50-bc571de0c55f-kube-api-access-mkcdn" pod "41cb130e-a0d3-44dc-9e50-bc571de0c55f" (UID: "41cb130e-a0d3-44dc-9e50-bc571de0c55f"). InnerVolumeSpecName "kube-api-access-mkcdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:23.865838 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:23.865803 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkcdn\" (UniqueName: \"kubernetes.io/projected/41cb130e-a0d3-44dc-9e50-bc571de0c55f-kube-api-access-mkcdn\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:23.865838 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:23.865834 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41cb130e-a0d3-44dc-9e50-bc571de0c55f-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:24.470969 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:24.470932 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" event={"ID":"41cb130e-a0d3-44dc-9e50-bc571de0c55f","Type":"ContainerDied","Data":"9e8135a5718dee06f0fa75d261a0b39043f99b692b6741f68f8c34b5db728344"} Oct 31 13:27:24.470969 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:24.470963 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8" Oct 31 13:27:24.471177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:24.470984 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e8135a5718dee06f0fa75d261a0b39043f99b692b6741f68f8c34b5db728344" Oct 31 13:27:28.377639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.377587 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272"] Oct 31 13:27:28.378260 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.378241 5532 cpu_manager.go:414] "containerMap: removing container" podUID="41cb130e-a0d3-44dc-9e50-bc571de0c55f" containerName="revoke-keys" Oct 31 13:27:28.378260 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.378260 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cb130e-a0d3-44dc-9e50-bc571de0c55f" containerName="revoke-keys" Oct 31 13:27:28.378398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.378345 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cb130e-a0d3-44dc-9e50-bc571de0c55f" containerName="revoke-keys" Oct 31 13:27:28.382100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.382081 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" Oct 31 13:27:28.394123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.394094 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272"] Oct 31 13:27:28.452823 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.452798 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:27:28.453072 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.453055 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:27:28.534407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.534372 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9778441d-bq272\" (UID: \"4121a829-90af-4108-ac5c-d2aef1e797c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" Oct 31 13:27:28.534577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.534428 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9778441d-bq272\" (UID: \"4121a829-90af-4108-ac5c-d2aef1e797c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" Oct 31 13:27:28.534654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.534582 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-462dt\" (UniqueName: \"kubernetes.io/projected/4121a829-90af-4108-ac5c-d2aef1e797c8-kube-api-access-462dt\") pod \"maas-api-revoke-sub-9778441d-bq272\" (UID: \"4121a829-90af-4108-ac5c-d2aef1e797c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" Oct 31 13:27:28.624359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.624323 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t"] Oct 31 13:27:28.628650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.628540 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.635476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.635443 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-462dt\" (UniqueName: \"kubernetes.io/projected/4121a829-90af-4108-ac5c-d2aef1e797c8-kube-api-access-462dt\") pod \"maas-api-revoke-sub-9778441d-bq272\" (UID: \"4121a829-90af-4108-ac5c-d2aef1e797c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" Oct 31 13:27:28.635720 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.635637 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9778441d-bq272\" (UID: \"4121a829-90af-4108-ac5c-d2aef1e797c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" Oct 31 13:27:28.635720 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.635705 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9778441d-bq272\" (UID: \"4121a829-90af-4108-ac5c-d2aef1e797c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" Oct 31 13:27:28.636536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.636496 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9778441d-bq272\" (UID: \"4121a829-90af-4108-ac5c-d2aef1e797c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" Oct 31 13:27:28.638720 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.638695 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9778441d-bq272\" (UID: \"4121a829-90af-4108-ac5c-d2aef1e797c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" Oct 31 13:27:28.652699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.652671 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t"] Oct 31 13:27:28.656182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.656153 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-462dt\" (UniqueName: \"kubernetes.io/projected/4121a829-90af-4108-ac5c-d2aef1e797c8-kube-api-access-462dt\") pod \"maas-api-revoke-sub-9778441d-bq272\" (UID: \"4121a829-90af-4108-ac5c-d2aef1e797c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" Oct 31 13:27:28.682501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.682463 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85"] Oct 31 13:27:28.687685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.687661 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.696127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.696090 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85"] Oct 31 13:27:28.700846 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.700825 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" Oct 31 13:27:28.719604 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.719568 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-aabeef-openshift-default-dockercfg-wlnkc\"" Oct 31 13:27:28.737342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.737298 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.737511 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.737489 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.737575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.737543 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.737796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.737772 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.737853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.737817 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.737973 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.737956 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.738018 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.738005 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27hd2\" (UniqueName: \"kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-kube-api-access-27hd2\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.738124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.738107 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.738210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.738176 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.767078 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.767053 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc"] Oct 31 13:27:28.773480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.773452 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.781973 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.781942 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc"] Oct 31 13:27:28.819138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.819108 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5286ea-openshift-default-dockercfg-rsjlf\"" Oct 31 13:27:28.839148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839112 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.839297 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839153 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.839297 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839178 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.839417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839333 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.839417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839395 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.839528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839429 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.839528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839467 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.839528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839512 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.839673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839642 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.839728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839696 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqrmh\" (UniqueName: \"kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-kube-api-access-dqrmh\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.839790 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839776 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.839827 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839799 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.839827 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839819 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.839920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839858 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.839920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839890 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.840028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.839950 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.840028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840004 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.840135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840037 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.840135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840077 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27hd2\" (UniqueName: \"kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-kube-api-access-27hd2\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.840135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840110 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.840135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840126 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.840342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840144 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.840342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840133 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.840342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840199 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.840342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840228 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwgll\" (UniqueName: \"kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-kube-api-access-gwgll\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.840342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840236 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.840342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840264 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.840342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840292 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.840342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840330 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.840772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840400 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.840772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840529 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.840772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.840759 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.842201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.842175 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.842304 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.842248 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.853206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.853186 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.853470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.853439 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27hd2\" (UniqueName: \"kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-kube-api-access-27hd2\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.941817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.941786 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.942006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.941831 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.942006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.941853 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.942006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.941875 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqrmh\" (UniqueName: \"kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-kube-api-access-dqrmh\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.942006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.941901 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.942006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.941932 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.942006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.941960 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.942006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.941992 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.942379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942034 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.942379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942134 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwgll\" (UniqueName: \"kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-kube-api-access-gwgll\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.942379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942161 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.942379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942185 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.942379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942207 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.942379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942246 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.942379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942300 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.942379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942319 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.942379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942373 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.942851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942403 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.942851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942442 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.942851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942478 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.942851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942501 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.942851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942619 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.942851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942624 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.942851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.942718 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.943234 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.943010 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.943234 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.943182 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.943338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.943286 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.943393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.943340 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.944642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.944584 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.944767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.944735 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.944942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.944923 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.945091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.945075 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.948195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.948179 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:28.958036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.958008 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.958120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.958039 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwgll\" (UniqueName: \"kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-kube-api-access-gwgll\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:28.958120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.958101 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqrmh\" (UniqueName: \"kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-kube-api-access-dqrmh\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.958449 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.958433 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:28.990237 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:28.990201 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4121a829_90af_4108_ac5c_d2aef1e797c8.slice/crio-3cd04ffd2ea23696514e9f097239580b47eed2feb1e25f7abb9bfb847a04e859 WatchSource:0}: Error finding container 3cd04ffd2ea23696514e9f097239580b47eed2feb1e25f7abb9bfb847a04e859: Status 404 returned error can't find the container with id 3cd04ffd2ea23696514e9f097239580b47eed2feb1e25f7abb9bfb847a04e859 Oct 31 13:27:28.990786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:28.990720 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272"] Oct 31 13:27:29.007179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.007015 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:29.105897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.105758 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:29.238772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.238738 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t"] Oct 31 13:27:29.239382 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:29.239350 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29ed7132_d493_4fc3_bc3b_b9e21fb3ebcc.slice/crio-fcc4cff8634c8a4d6b95c1a2b974aef6f5da4f308aea059ebb7a9cc8b8333532 WatchSource:0}: Error finding container fcc4cff8634c8a4d6b95c1a2b974aef6f5da4f308aea059ebb7a9cc8b8333532: Status 404 returned error can't find the container with id fcc4cff8634c8a4d6b95c1a2b974aef6f5da4f308aea059ebb7a9cc8b8333532 Oct 31 13:27:29.242063 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.242033 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:29.242149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.242103 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:29.242149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.242135 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:29.308241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.308217 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85"] Oct 31 13:27:29.308929 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:29.308903 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb572ef9e_7dc1_4886_bfa4_ec84b98dee31.slice/crio-336448b0a4f630e02668ef7e1ca83ba4df808f541fd20edcd6d0133a0ab23533 WatchSource:0}: Error finding container 336448b0a4f630e02668ef7e1ca83ba4df808f541fd20edcd6d0133a0ab23533: Status 404 returned error can't find the container with id 336448b0a4f630e02668ef7e1ca83ba4df808f541fd20edcd6d0133a0ab23533 Oct 31 13:27:29.311514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.311487 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:29.311582 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.311550 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:29.311647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.311579 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:29.397582 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.397553 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc"] Oct 31 13:27:29.399058 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:29.399029 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod226dd9d4_80bc_4c59_8f41_d1cefc341cf9.slice/crio-bbdb756b8338fb1ac99956345f57ea65474c7e32673543d0a8336c165ff30613 WatchSource:0}: Error finding container bbdb756b8338fb1ac99956345f57ea65474c7e32673543d0a8336c165ff30613: Status 404 returned error can't find the container with id bbdb756b8338fb1ac99956345f57ea65474c7e32673543d0a8336c165ff30613 Oct 31 13:27:29.401776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.401750 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:29.401834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.401822 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:29.401873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.401848 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:29.533685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.533568 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" event={"ID":"226dd9d4-80bc-4c59-8f41-d1cefc341cf9","Type":"ContainerStarted","Data":"e71e671ac5d840b7f6ef1b60dfa465bbf836bbfc201d6a62984d1c04851f1017"} Oct 31 13:27:29.533685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.533642 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" event={"ID":"226dd9d4-80bc-4c59-8f41-d1cefc341cf9","Type":"ContainerStarted","Data":"bbdb756b8338fb1ac99956345f57ea65474c7e32673543d0a8336c165ff30613"} Oct 31 13:27:29.536700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.536671 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" event={"ID":"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc","Type":"ContainerStarted","Data":"786a4572ea4d476e76dc01c7664bf4201c7aaf018646b40999b52b7526750407"} Oct 31 13:27:29.536854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.536722 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" event={"ID":"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc","Type":"ContainerStarted","Data":"fcc4cff8634c8a4d6b95c1a2b974aef6f5da4f308aea059ebb7a9cc8b8333532"} Oct 31 13:27:29.539308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.538824 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" event={"ID":"b572ef9e-7dc1-4886-bfa4-ec84b98dee31","Type":"ContainerStarted","Data":"9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa"} Oct 31 13:27:29.539308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.538864 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" event={"ID":"b572ef9e-7dc1-4886-bfa4-ec84b98dee31","Type":"ContainerStarted","Data":"336448b0a4f630e02668ef7e1ca83ba4df808f541fd20edcd6d0133a0ab23533"} Oct 31 13:27:29.542494 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.542433 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" event={"ID":"4121a829-90af-4108-ac5c-d2aef1e797c8","Type":"ContainerStarted","Data":"c0ba86706adfcc24dbcc1695a1e50260153ecee2d65f108dd30ad5dfcd6bb73e"} Oct 31 13:27:29.542494 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.542472 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" event={"ID":"4121a829-90af-4108-ac5c-d2aef1e797c8","Type":"ContainerStarted","Data":"3cd04ffd2ea23696514e9f097239580b47eed2feb1e25f7abb9bfb847a04e859"} Oct 31 13:27:29.562612 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.562530 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" podStartSLOduration=1.562511535 podStartE2EDuration="1.562511535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:29.561968132 +0000 UTC m=+1417.707711962" watchObservedRunningTime="2026-10-31 13:27:29.562511535 +0000 UTC m=+1417.708255366" Oct 31 13:27:29.622343 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.622291 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" podStartSLOduration=1.622275747 podStartE2EDuration="1.622275747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:29.620405433 +0000 UTC m=+1417.766149261" watchObservedRunningTime="2026-10-31 13:27:29.622275747 +0000 UTC m=+1417.768019645" Oct 31 13:27:29.623502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.623463 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" podStartSLOduration=1.623451679 podStartE2EDuration="1.623451679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:29.594632078 +0000 UTC m=+1417.740375907" watchObservedRunningTime="2026-10-31 13:27:29.623451679 +0000 UTC m=+1417.769195506" Oct 31 13:27:29.654314 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.654246 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" podStartSLOduration=1.654223844 podStartE2EDuration="1.654223844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:29.649644655 +0000 UTC m=+1417.795388484" watchObservedRunningTime="2026-10-31 13:27:29.654223844 +0000 UTC m=+1417.799967689" Oct 31 13:27:29.949288 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:29.949260 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:30.007744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:30.007715 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:30.009688 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:30.009656 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 13:27:30.009798 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:30.009715 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 13:27:30.106951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:30.106915 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:30.108306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:30.108278 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 13:27:30.108422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:30.108335 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" podUID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 13:27:30.579672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:30.579635 5532 generic.go:356] "Generic (PLEG): container finished" podID="4121a829-90af-4108-ac5c-d2aef1e797c8" containerID="c0ba86706adfcc24dbcc1695a1e50260153ecee2d65f108dd30ad5dfcd6bb73e" exitCode=0 Oct 31 13:27:30.587003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:30.586032 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" event={"ID":"4121a829-90af-4108-ac5c-d2aef1e797c8","Type":"ContainerDied","Data":"c0ba86706adfcc24dbcc1695a1e50260153ecee2d65f108dd30ad5dfcd6bb73e"} Oct 31 13:27:30.964616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:30.954184 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 13:27:30.964616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:30.954262 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" podUID="29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 13:27:31.008620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:31.008008 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 13:27:31.008620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:31.008071 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 13:27:31.112093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:31.112048 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 13:27:31.112409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:31.112371 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" podUID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 13:27:31.953839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:31.953802 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 13:27:32.023561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:31.966864 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" podUID="29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 13:27:32.023561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.022377 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 13:27:32.023561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.023103 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 13:27:32.025254 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:27:32.025130 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:27:32.109618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.107078 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 13:27:32.109618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.107141 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" podUID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 13:27:32.265617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.264767 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" Oct 31 13:27:32.289620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.272260 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-openshift-ca\") pod \"4121a829-90af-4108-ac5c-d2aef1e797c8\" (UID: \"4121a829-90af-4108-ac5c-d2aef1e797c8\") " Oct 31 13:27:32.289620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.272357 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4121a829-90af-4108-ac5c-d2aef1e797c8-kube-api-access-462dt\" (UniqueName: \"kubernetes.io/projected/4121a829-90af-4108-ac5c-d2aef1e797c8-kube-api-access-462dt\") pod \"4121a829-90af-4108-ac5c-d2aef1e797c8\" (UID: \"4121a829-90af-4108-ac5c-d2aef1e797c8\") " Oct 31 13:27:32.289620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.272448 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-xks-ca\") pod \"4121a829-90af-4108-ac5c-d2aef1e797c8\" (UID: \"4121a829-90af-4108-ac5c-d2aef1e797c8\") " Oct 31 13:27:32.289620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.279432 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-openshift-ca" pod "4121a829-90af-4108-ac5c-d2aef1e797c8" (UID: "4121a829-90af-4108-ac5c-d2aef1e797c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:32.310616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.303787 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4121a829-90af-4108-ac5c-d2aef1e797c8-kube-api-access-462dt" pod "4121a829-90af-4108-ac5c-d2aef1e797c8" (UID: "4121a829-90af-4108-ac5c-d2aef1e797c8"). InnerVolumeSpecName "kube-api-access-462dt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:32.333636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.330954 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-xks-ca" pod "4121a829-90af-4108-ac5c-d2aef1e797c8" (UID: "4121a829-90af-4108-ac5c-d2aef1e797c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:27:32.377620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.377491 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:32.377620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.377530 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-462dt\" (UniqueName: \"kubernetes.io/projected/4121a829-90af-4108-ac5c-d2aef1e797c8-kube-api-access-462dt\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:32.377620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.377546 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4121a829-90af-4108-ac5c-d2aef1e797c8-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:32.628276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.628191 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" event={"ID":"4121a829-90af-4108-ac5c-d2aef1e797c8","Type":"ContainerDied","Data":"3cd04ffd2ea23696514e9f097239580b47eed2feb1e25f7abb9bfb847a04e859"} Oct 31 13:27:32.628276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.628241 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cd04ffd2ea23696514e9f097239580b47eed2feb1e25f7abb9bfb847a04e859" Oct 31 13:27:32.628464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.628290 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272" Oct 31 13:27:32.955618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.951970 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 13:27:32.955618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:32.952035 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" podUID="29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 13:27:33.010629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:33.009996 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 13:27:33.010629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:33.010057 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 13:27:33.107618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:33.107510 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 13:27:33.107618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:33.107570 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" podUID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 13:27:33.960620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:33.956992 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 13:27:33.960620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:33.957077 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" podUID="29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 13:27:34.035645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:34.018973 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 13:27:34.035645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:34.019041 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 13:27:34.118622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:34.118515 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 13:27:34.118622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:34.118575 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" podUID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 13:27:34.956618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:34.953138 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 13:27:34.956618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:34.953216 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" podUID="29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 13:27:35.010620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:35.008979 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 13:27:35.010620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:35.009040 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 13:27:35.112621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:35.112548 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 13:27:35.112830 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:35.112626 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" podUID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 13:27:35.949236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:35.949195 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Oct 31 13:27:35.949390 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:35.949264 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" podUID="29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Oct 31 13:27:36.007774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:36.007735 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 13:27:36.007938 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:36.007805 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 13:27:36.106753 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:36.106714 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Oct 31 13:27:36.107141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:36.106777 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" podUID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Oct 31 13:27:37.004840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:37.004798 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:37.005751 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:37.005725 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:37.007933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:37.007907 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:27:37.017012 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:37.016985 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:37.017677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:37.017618 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:37.018925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:37.018903 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:27:37.112834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:37.112799 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:37.113987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:37.113964 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:37.115259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:37.115242 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:27:39.114326 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.114282 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k"] Oct 31 13:27:39.114880 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.114862 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4121a829-90af-4108-ac5c-d2aef1e797c8" containerName="revoke-keys" Oct 31 13:27:39.114927 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.114881 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4121a829-90af-4108-ac5c-d2aef1e797c8" containerName="revoke-keys" Oct 31 13:27:39.115019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.115010 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4121a829-90af-4108-ac5c-d2aef1e797c8" containerName="revoke-keys" Oct 31 13:27:39.118465 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.118447 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" Oct 31 13:27:39.125525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.125502 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:27:39.125983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.125627 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:27:39.128365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.128344 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k"] Oct 31 13:27:39.229206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.229173 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22r66\" (UniqueName: \"kubernetes.io/projected/f16a2544-082d-465a-86cc-709a38c3adc1-kube-api-access-22r66\") pod \"maas-api-revoke-sub-f9812a60-xp55k\" (UID: \"f16a2544-082d-465a-86cc-709a38c3adc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" Oct 31 13:27:39.229368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.229211 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9812a60-xp55k\" (UID: \"f16a2544-082d-465a-86cc-709a38c3adc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" Oct 31 13:27:39.229412 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.229383 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9812a60-xp55k\" (UID: \"f16a2544-082d-465a-86cc-709a38c3adc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" Oct 31 13:27:39.330918 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.330884 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9812a60-xp55k\" (UID: \"f16a2544-082d-465a-86cc-709a38c3adc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" Oct 31 13:27:39.331088 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.331051 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22r66\" (UniqueName: \"kubernetes.io/projected/f16a2544-082d-465a-86cc-709a38c3adc1-kube-api-access-22r66\") pod \"maas-api-revoke-sub-f9812a60-xp55k\" (UID: \"f16a2544-082d-465a-86cc-709a38c3adc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" Oct 31 13:27:39.331175 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.331085 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9812a60-xp55k\" (UID: \"f16a2544-082d-465a-86cc-709a38c3adc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" Oct 31 13:27:39.331749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.331720 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f9812a60-xp55k\" (UID: \"f16a2544-082d-465a-86cc-709a38c3adc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" Oct 31 13:27:39.333260 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.333240 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f9812a60-xp55k\" (UID: \"f16a2544-082d-465a-86cc-709a38c3adc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" Oct 31 13:27:39.345011 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.344982 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22r66\" (UniqueName: \"kubernetes.io/projected/f16a2544-082d-465a-86cc-709a38c3adc1-kube-api-access-22r66\") pod \"maas-api-revoke-sub-f9812a60-xp55k\" (UID: \"f16a2544-082d-465a-86cc-709a38c3adc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" Oct 31 13:27:39.433503 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.433469 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" Oct 31 13:27:39.716789 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.716753 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k"] Oct 31 13:27:39.717960 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:39.717935 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf16a2544_082d_465a_86cc_709a38c3adc1.slice/crio-d8f7d20c73d9d80ac477adce0cad5e76ffa9f58241bcf92b613bad9a5340eedd WatchSource:0}: Error finding container d8f7d20c73d9d80ac477adce0cad5e76ffa9f58241bcf92b613bad9a5340eedd: Status 404 returned error can't find the container with id d8f7d20c73d9d80ac477adce0cad5e76ffa9f58241bcf92b613bad9a5340eedd Oct 31 13:27:39.755138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:39.755107 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" event={"ID":"f16a2544-082d-465a-86cc-709a38c3adc1","Type":"ContainerStarted","Data":"d8f7d20c73d9d80ac477adce0cad5e76ffa9f58241bcf92b613bad9a5340eedd"} Oct 31 13:27:40.762635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:40.762574 5532 generic.go:356] "Generic (PLEG): container finished" podID="f16a2544-082d-465a-86cc-709a38c3adc1" containerID="73f4b6d65df5bdbe79bb3ecd33305e7efa04cd0c8809da9abe3a7aa38c3339d5" exitCode=0 Oct 31 13:27:40.762635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:40.762623 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" event={"ID":"f16a2544-082d-465a-86cc-709a38c3adc1","Type":"ContainerDied","Data":"73f4b6d65df5bdbe79bb3ecd33305e7efa04cd0c8809da9abe3a7aa38c3339d5"} Oct 31 13:27:41.903641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:41.903614 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" Oct 31 13:27:41.963712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:41.963681 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-openshift-ca\") pod \"f16a2544-082d-465a-86cc-709a38c3adc1\" (UID: \"f16a2544-082d-465a-86cc-709a38c3adc1\") " Oct 31 13:27:41.963931 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:41.963912 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f16a2544-082d-465a-86cc-709a38c3adc1-kube-api-access-22r66\" (UniqueName: \"kubernetes.io/projected/f16a2544-082d-465a-86cc-709a38c3adc1-kube-api-access-22r66\") pod \"f16a2544-082d-465a-86cc-709a38c3adc1\" (UID: \"f16a2544-082d-465a-86cc-709a38c3adc1\") " Oct 31 13:27:41.963993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:41.963977 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-xks-ca\") pod \"f16a2544-082d-465a-86cc-709a38c3adc1\" (UID: \"f16a2544-082d-465a-86cc-709a38c3adc1\") " Oct 31 13:27:41.964046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:41.964017 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-openshift-ca" pod "f16a2544-082d-465a-86cc-709a38c3adc1" (UID: "f16a2544-082d-465a-86cc-709a38c3adc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:41.964520 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:41.964490 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:41.966018 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:41.965987 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-xks-ca" pod "f16a2544-082d-465a-86cc-709a38c3adc1" (UID: "f16a2544-082d-465a-86cc-709a38c3adc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:27:41.966162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:41.966089 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f16a2544-082d-465a-86cc-709a38c3adc1-kube-api-access-22r66" pod "f16a2544-082d-465a-86cc-709a38c3adc1" (UID: "f16a2544-082d-465a-86cc-709a38c3adc1"). InnerVolumeSpecName "kube-api-access-22r66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:42.065487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:42.065423 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22r66\" (UniqueName: \"kubernetes.io/projected/f16a2544-082d-465a-86cc-709a38c3adc1-kube-api-access-22r66\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:42.065487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:42.065450 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f16a2544-082d-465a-86cc-709a38c3adc1-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:42.776493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:42.776462 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" Oct 31 13:27:42.776667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:42.776496 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k" event={"ID":"f16a2544-082d-465a-86cc-709a38c3adc1","Type":"ContainerDied","Data":"d8f7d20c73d9d80ac477adce0cad5e76ffa9f58241bcf92b613bad9a5340eedd"} Oct 31 13:27:42.776667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:42.776547 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8f7d20c73d9d80ac477adce0cad5e76ffa9f58241bcf92b613bad9a5340eedd" Oct 31 13:27:44.602708 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:44.602666 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:27:49.640722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.640645 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq"] Oct 31 13:27:49.641265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.641249 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f16a2544-082d-465a-86cc-709a38c3adc1" containerName="revoke-keys" Oct 31 13:27:49.641312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.641267 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f16a2544-082d-465a-86cc-709a38c3adc1" containerName="revoke-keys" Oct 31 13:27:49.641393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.641383 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f16a2544-082d-465a-86cc-709a38c3adc1" containerName="revoke-keys" Oct 31 13:27:49.645996 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.645980 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.651981 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.651958 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8b2f0a\"/\"kube-root-ca.crt\"" Oct 31 13:27:49.653748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.653726 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-8b2f0a\"/\"openshift-service-ca.crt\"" Oct 31 13:27:49.653964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.653855 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8b2f0a\"/\"default-dockercfg-mxnz2\"" Oct 31 13:27:49.654031 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.653967 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-8b2f0a\"/\"distinct-2-w4-8b2f0a-kserve-self-signed-certs\"" Oct 31 13:27:49.656933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.656911 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq"] Oct 31 13:27:49.777647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.777588 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-dshm\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.777986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.777958 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tmp-dir\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.779086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.779057 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n48g\" (UniqueName: \"kubernetes.io/projected/27dbfb95-a109-4e31-b1a0-4ac435c252d3-kube-api-access-8n48g\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.779425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.779402 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-home\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.779572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.779555 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-model-cache\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.779709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.779693 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tls-certs\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.881446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.881413 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-home\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.881714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.881451 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-model-cache\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.881714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.881477 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tls-certs\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.881714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.881522 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-dshm\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.881714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.881605 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tmp-dir\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.881714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.881659 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n48g\" (UniqueName: \"kubernetes.io/projected/27dbfb95-a109-4e31-b1a0-4ac435c252d3-kube-api-access-8n48g\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.882002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.881909 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-model-cache\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.882092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.882001 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-home\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.882154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.882135 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tmp-dir\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.883867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.883845 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-dshm\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.884264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.884241 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tls-certs\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.895958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.895932 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n48g\" (UniqueName: \"kubernetes.io/projected/27dbfb95-a109-4e31-b1a0-4ac435c252d3-kube-api-access-8n48g\") pod \"distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:49.960393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:49.960309 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:50.247956 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:50.247920 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27dbfb95_a109_4e31_b1a0_4ac435c252d3.slice/crio-0ecf6b89f9a9172c2079b4da536c1f2b2b75af47c2b6e52d04b2a509e5b644f8 WatchSource:0}: Error finding container 0ecf6b89f9a9172c2079b4da536c1f2b2b75af47c2b6e52d04b2a509e5b644f8: Status 404 returned error can't find the container with id 0ecf6b89f9a9172c2079b4da536c1f2b2b75af47c2b6e52d04b2a509e5b644f8 Oct 31 13:27:50.248351 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:50.248328 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq"] Oct 31 13:27:50.847787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:50.847748 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" event={"ID":"27dbfb95-a109-4e31-b1a0-4ac435c252d3","Type":"ContainerStarted","Data":"916beffa560feb0154ae2c082af496056624a6f1f28de9f8e8edfcc08c86a93b"} Oct 31 13:27:50.848226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:50.847797 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" event={"ID":"27dbfb95-a109-4e31-b1a0-4ac435c252d3","Type":"ContainerStarted","Data":"0ecf6b89f9a9172c2079b4da536c1f2b2b75af47c2b6e52d04b2a509e5b644f8"} Oct 31 13:27:50.887797 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:50.887741 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podStartSLOduration=1.8877271 podStartE2EDuration="1.8877271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:50.886687195 +0000 UTC m=+1439.032431023" watchObservedRunningTime="2026-10-31 13:27:50.8877271 +0000 UTC m=+1439.033470927" Oct 31 13:27:50.961089 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:50.961059 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:51.216371 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.216339 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s"] Oct 31 13:27:51.220440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.220424 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" Oct 31 13:27:51.227098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.227075 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:27:51.227098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.227084 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:27:51.232276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.232255 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s"] Oct 31 13:27:51.300386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.300351 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qd7p\" (UniqueName: \"kubernetes.io/projected/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-kube-api-access-5qd7p\") pod \"maas-api-revoke-sub-737725b5-ph78s\" (UID: \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" Oct 31 13:27:51.300544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.300397 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-737725b5-ph78s\" (UID: \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" Oct 31 13:27:51.300638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.300625 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-737725b5-ph78s\" (UID: \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" Oct 31 13:27:51.401492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.401461 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-737725b5-ph78s\" (UID: \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" Oct 31 13:27:51.401675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.401528 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qd7p\" (UniqueName: \"kubernetes.io/projected/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-kube-api-access-5qd7p\") pod \"maas-api-revoke-sub-737725b5-ph78s\" (UID: \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" Oct 31 13:27:51.401675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.401558 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-737725b5-ph78s\" (UID: \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" Oct 31 13:27:51.402263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.402239 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-737725b5-ph78s\" (UID: \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" Oct 31 13:27:51.403849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.403829 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-737725b5-ph78s\" (UID: \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" Oct 31 13:27:51.415437 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.415416 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qd7p\" (UniqueName: \"kubernetes.io/projected/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-kube-api-access-5qd7p\") pod \"maas-api-revoke-sub-737725b5-ph78s\" (UID: \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" Oct 31 13:27:51.536275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.536195 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" Oct 31 13:27:51.822678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.821069 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s"] Oct 31 13:27:51.824430 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:51.824398 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea0fbcc_2d0c_415a_a5cf_e3f8075737de.slice/crio-97ebce5cfd9e585e122c1bf650704d29ec3966f90962f97a8a8ee1065d18501f WatchSource:0}: Error finding container 97ebce5cfd9e585e122c1bf650704d29ec3966f90962f97a8a8ee1065d18501f: Status 404 returned error can't find the container with id 97ebce5cfd9e585e122c1bf650704d29ec3966f90962f97a8a8ee1065d18501f Oct 31 13:27:51.855253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:51.855220 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" event={"ID":"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de","Type":"ContainerStarted","Data":"97ebce5cfd9e585e122c1bf650704d29ec3966f90962f97a8a8ee1065d18501f"} Oct 31 13:27:52.862999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:52.862958 5532 generic.go:356] "Generic (PLEG): container finished" podID="3ea0fbcc-2d0c-415a-a5cf-e3f8075737de" containerID="0f41719e68eb2c25a6a14568dc35914b83041679f6c1f66b6131d392c9bd3d2b" exitCode=0 Oct 31 13:27:52.863468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:52.863040 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" event={"ID":"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de","Type":"ContainerDied","Data":"0f41719e68eb2c25a6a14568dc35914b83041679f6c1f66b6131d392c9bd3d2b"} Oct 31 13:27:54.012324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:54.012297 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" Oct 31 13:27:54.138294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:54.138207 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-kube-api-access-5qd7p\" (UniqueName: \"kubernetes.io/projected/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-kube-api-access-5qd7p\") pod \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\" (UID: \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\") " Oct 31 13:27:54.138294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:54.138281 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-xks-ca\") pod \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\" (UID: \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\") " Oct 31 13:27:54.138528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:54.138343 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-openshift-ca\") pod \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\" (UID: \"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de\") " Oct 31 13:27:54.138763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:54.138735 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-openshift-ca" pod "3ea0fbcc-2d0c-415a-a5cf-e3f8075737de" (UID: "3ea0fbcc-2d0c-415a-a5cf-e3f8075737de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:27:54.140353 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:54.140319 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-xks-ca" pod "3ea0fbcc-2d0c-415a-a5cf-e3f8075737de" (UID: "3ea0fbcc-2d0c-415a-a5cf-e3f8075737de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:27:54.140450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:54.140390 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-kube-api-access-5qd7p" pod "3ea0fbcc-2d0c-415a-a5cf-e3f8075737de" (UID: "3ea0fbcc-2d0c-415a-a5cf-e3f8075737de"). InnerVolumeSpecName "kube-api-access-5qd7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:27:54.240133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:54.240101 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:54.240133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:54.240130 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:54.240317 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:54.240142 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qd7p\" (UniqueName: \"kubernetes.io/projected/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de-kube-api-access-5qd7p\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:27:54.876403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:54.876377 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" Oct 31 13:27:54.876563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:54.876407 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s" event={"ID":"3ea0fbcc-2d0c-415a-a5cf-e3f8075737de","Type":"ContainerDied","Data":"97ebce5cfd9e585e122c1bf650704d29ec3966f90962f97a8a8ee1065d18501f"} Oct 31 13:27:54.876563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:54.876463 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="97ebce5cfd9e585e122c1bf650704d29ec3966f90962f97a8a8ee1065d18501f" Oct 31 13:27:58.551260 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.551222 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr"] Oct 31 13:27:58.551813 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.551798 5532 cpu_manager.go:414] "containerMap: removing container" podUID="3ea0fbcc-2d0c-415a-a5cf-e3f8075737de" containerName="revoke-keys" Oct 31 13:27:58.551872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.551813 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ea0fbcc-2d0c-415a-a5cf-e3f8075737de" containerName="revoke-keys" Oct 31 13:27:58.551934 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.551925 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ea0fbcc-2d0c-415a-a5cf-e3f8075737de" containerName="revoke-keys" Oct 31 13:27:58.555645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.555628 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.570397 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.570367 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr"] Oct 31 13:27:58.697531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.697497 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.697739 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.697556 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.697739 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.697588 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.697831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.697728 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.697831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.697825 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnlvc\" (UniqueName: \"kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-kube-api-access-vnlvc\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.697901 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.697869 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.697956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.697925 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48476992-f75d-4caf-aed3-02cd8c339644-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.698060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.698045 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48476992-f75d-4caf-aed3-02cd8c339644-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.698122 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.698088 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.799729 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.799689 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48476992-f75d-4caf-aed3-02cd8c339644-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.799901 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.799745 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.799901 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.799807 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.799901 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.799866 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.800083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.799909 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.800083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.799940 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.800083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.800049 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnlvc\" (UniqueName: \"kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-kube-api-access-vnlvc\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.800237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.800118 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.800237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.800156 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48476992-f75d-4caf-aed3-02cd8c339644-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.801035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.800640 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.801035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.800953 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.801035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.800999 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.801035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.801009 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48476992-f75d-4caf-aed3-02cd8c339644-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.801570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.801490 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.803311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.803264 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48476992-f75d-4caf-aed3-02cd8c339644-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.804824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.804803 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.814869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.814843 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnlvc\" (UniqueName: \"kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-kube-api-access-vnlvc\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.815126 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.815104 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:58.870492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:58.870451 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:27:59.183491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:59.183459 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr"] Oct 31 13:27:59.184086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:27:59.184053 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48476992_f75d_4caf_aed3_02cd8c339644.slice/crio-3ffb688c1cf744a20b3a29969f711b581cf29eb3521e4f3530ffcf92d97742d5 WatchSource:0}: Error finding container 3ffb688c1cf744a20b3a29969f711b581cf29eb3521e4f3530ffcf92d97742d5: Status 404 returned error can't find the container with id 3ffb688c1cf744a20b3a29969f711b581cf29eb3521e4f3530ffcf92d97742d5 Oct 31 13:27:59.190621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:59.187547 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:59.190621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:59.187723 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:59.190621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:59.187812 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:27:59.910351 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:59.910264 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" event={"ID":"48476992-f75d-4caf-aed3-02cd8c339644","Type":"ContainerStarted","Data":"40899891746f94dbd718f4c325645ca1ebfa0bb51a2ee4435634eb6ba4679847"} Oct 31 13:27:59.910761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:59.910358 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" event={"ID":"48476992-f75d-4caf-aed3-02cd8c339644","Type":"ContainerStarted","Data":"3ffb688c1cf744a20b3a29969f711b581cf29eb3521e4f3530ffcf92d97742d5"} Oct 31 13:27:59.938694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:59.938634 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" podStartSLOduration=1.9386184370000001 podStartE2EDuration="1.938618437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:27:59.936618111 +0000 UTC m=+1448.082361943" watchObservedRunningTime="2026-10-31 13:27:59.938618437 +0000 UTC m=+1448.084362262" Oct 31 13:27:59.966956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:59.964828 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:27:59.978024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:27:59.977996 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:28:00.870583 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:00.870548 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:28:00.872496 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:00.872470 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:28:00.872646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:00.872530 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" podUID="48476992-f75d-4caf-aed3-02cd8c339644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 13:28:00.927697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:00.927669 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:28:01.784732 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:28:01.784690 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:28:01.871704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:01.871656 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:28:01.871931 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:01.871720 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" podUID="48476992-f75d-4caf-aed3-02cd8c339644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 13:28:02.870924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:02.870884 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:28:02.871349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:02.870960 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" podUID="48476992-f75d-4caf-aed3-02cd8c339644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 13:28:03.871262 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:03.871230 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:28:03.871721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:03.871282 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" podUID="48476992-f75d-4caf-aed3-02cd8c339644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 13:28:04.871278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:04.871243 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:28:04.871673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:04.871301 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" podUID="48476992-f75d-4caf-aed3-02cd8c339644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 13:28:05.871449 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:05.871412 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Oct 31 13:28:05.871992 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:05.871480 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" podUID="48476992-f75d-4caf-aed3-02cd8c339644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Oct 31 13:28:06.152334 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.148960 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww"] Oct 31 13:28:06.154534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.154509 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww"] Oct 31 13:28:06.155817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.155747 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.167538 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.166544 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c82140\"/\"distinct-w3-c82140-kserve-self-signed-certs\"" Oct 31 13:28:06.239885 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.239844 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-home\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.240631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.240587 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-tmp-dir\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.240853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.240831 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-dshm\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.241041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.241022 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxqdw\" (UniqueName: \"kubernetes.io/projected/09ab1bcc-1eac-475b-af39-23d43405e233-kube-api-access-hxqdw\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.241165 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.241149 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ab1bcc-1eac-475b-af39-23d43405e233-tls-certs\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.241368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.241351 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-model-cache\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.343802 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.343769 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-home\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.344709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.344683 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-tmp-dir\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.344932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.344913 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-dshm\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.345116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.345096 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxqdw\" (UniqueName: \"kubernetes.io/projected/09ab1bcc-1eac-475b-af39-23d43405e233-kube-api-access-hxqdw\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.345245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.345228 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ab1bcc-1eac-475b-af39-23d43405e233-tls-certs\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.345432 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.345415 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-model-cache\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.346264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.346243 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-model-cache\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.352700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.344498 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-home\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.352854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.349525 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-tmp-dir\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.368622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.360091 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-dshm\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.368622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.362682 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ab1bcc-1eac-475b-af39-23d43405e233-tls-certs\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.381621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.379473 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxqdw\" (UniqueName: \"kubernetes.io/projected/09ab1bcc-1eac-475b-af39-23d43405e233-kube-api-access-hxqdw\") pod \"distinct-w3-c82140-kserve-6b5f85765b-zp2ww\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:06.487670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:06.484846 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:07.081645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:07.081524 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww"] Oct 31 13:28:07.167893 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:28:07.167856 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09ab1bcc_1eac_475b_af39_23d43405e233.slice/crio-5365e510d99fcb9a167862e637721534fb864ca0502e5e62b21561d3585ff5d6 WatchSource:0}: Error finding container 5365e510d99fcb9a167862e637721534fb864ca0502e5e62b21561d3585ff5d6: Status 404 returned error can't find the container with id 5365e510d99fcb9a167862e637721534fb864ca0502e5e62b21561d3585ff5d6 Oct 31 13:28:07.961105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:07.961069 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:28:07.961383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:07.961352 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" podUID="48476992-f75d-4caf-aed3-02cd8c339644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:28:08.087321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:08.087281 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:28:08.087946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:08.087895 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" podUID="29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:28:08.093807 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:08.093780 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:28:08.095613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:08.095581 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:28:08.097161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:08.097137 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:28:08.135249 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:08.135208 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" event={"ID":"09ab1bcc-1eac-475b-af39-23d43405e233","Type":"ContainerStarted","Data":"07230cd7162ff00ec76cd832a1af6c9dcf3e7141c037c2303a60b0510752bb3e"} Oct 31 13:28:08.135425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:08.135257 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" event={"ID":"09ab1bcc-1eac-475b-af39-23d43405e233","Type":"ContainerStarted","Data":"5365e510d99fcb9a167862e637721534fb864ca0502e5e62b21561d3585ff5d6"} Oct 31 13:28:08.180963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:08.180898 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podStartSLOduration=3.180879384 podStartE2EDuration="3.180879384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:08.173005491 +0000 UTC m=+1456.318749320" watchObservedRunningTime="2026-10-31 13:28:08.180879384 +0000 UTC m=+1456.326623213" Oct 31 13:28:08.485202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:08.485170 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:13.776585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:13.776545 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r"] Oct 31 13:28:13.781638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:13.781583 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:13.799850 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:13.799824 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r"] Oct 31 13:28:13.917337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:13.917301 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:13.917500 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:13.917361 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:13.917500 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:13.917433 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7m7\" (UniqueName: \"kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-kube-api-access-gz7m7\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:13.917585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:13.917544 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:13.917632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:13.917612 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:13.917676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:13.917635 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:13.917719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:13.917705 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:13.917799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:13.917783 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:13.917853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:13.917839 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.018610 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.018573 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.018805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.018669 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.018805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.018715 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.018805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.018742 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7m7\" (UniqueName: \"kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-kube-api-access-gz7m7\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.018976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.018832 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.018976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.018889 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.018976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.018926 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.018976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.018965 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.019199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.019015 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.019199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.019063 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.019314 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.019206 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.019414 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.019388 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.019526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.019442 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.019825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.019799 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.021019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.020999 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.021360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.021336 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.034708 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.034644 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7m7\" (UniqueName: \"kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-kube-api-access-gz7m7\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.035673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.035650 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.097046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.097018 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:14.389291 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.389261 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r"] Oct 31 13:28:14.389934 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:28:14.389911 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f805bdf_0bd7_4ea9_8378_18f10dbc7d98.slice/crio-1fa4ce02200a5dbea91d586ee35625810d89dd2c51cfbee324d30e620c0661d8 WatchSource:0}: Error finding container 1fa4ce02200a5dbea91d586ee35625810d89dd2c51cfbee324d30e620c0661d8: Status 404 returned error can't find the container with id 1fa4ce02200a5dbea91d586ee35625810d89dd2c51cfbee324d30e620c0661d8 Oct 31 13:28:14.392462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.392423 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:14.392557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.392503 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:14.392557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:14.392544 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:15.191018 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:15.190978 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" event={"ID":"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98","Type":"ContainerStarted","Data":"009187c1a0f7e1005a9283a579bfe10caa6754c94ec8ccb43599fca073a4440b"} Oct 31 13:28:15.191444 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:15.191045 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" event={"ID":"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98","Type":"ContainerStarted","Data":"1fa4ce02200a5dbea91d586ee35625810d89dd2c51cfbee324d30e620c0661d8"} Oct 31 13:28:15.223320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:15.223269 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" podStartSLOduration=2.223253601 podStartE2EDuration="2.223253601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:15.219584162 +0000 UTC m=+1463.365327991" watchObservedRunningTime="2026-10-31 13:28:15.223253601 +0000 UTC m=+1463.368997428" Oct 31 13:28:16.097844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:16.097803 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:16.099097 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:16.099066 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 13:28:16.099250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:16.099117 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" podUID="7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 13:28:16.485764 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:16.485733 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:16.498832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:16.498807 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:17.098087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:17.098040 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 13:28:17.098304 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:17.098107 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" podUID="7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 13:28:17.214959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:17.214931 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:28:18.097900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:18.097861 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 13:28:18.098353 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:18.097926 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" podUID="7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 13:28:19.097668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:19.097634 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 13:28:19.097845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:19.097694 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" podUID="7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 13:28:20.097942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:20.097901 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 13:28:20.098393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:20.097983 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" podUID="7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 13:28:21.097570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:21.097518 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Oct 31 13:28:21.097831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:21.097632 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" podUID="7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Oct 31 13:28:22.102446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:22.102410 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:22.102931 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:22.102815 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:22.103524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:22.103500 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:28:26.712079 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:26.712045 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:28:28.564425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.564376 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt"] Oct 31 13:28:28.578951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.578923 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.618519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.618481 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9332ef-beec-4b46-9617-36d8ce8812c5-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.618805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.618780 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skbk4\" (UniqueName: \"kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-kube-api-access-skbk4\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.619000 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.618981 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.619223 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.619203 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.619382 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.619364 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.619541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.619524 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.619695 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.619676 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.619866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.619849 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.620028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.620011 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.701894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.701802 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt"] Oct 31 13:28:28.752775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.732128 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.753140 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.753107 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.753318 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.753296 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.753469 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.753453 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.759754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.759718 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.759971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.759952 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9332ef-beec-4b46-9617-36d8ce8812c5-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.760112 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.760094 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skbk4\" (UniqueName: \"kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-kube-api-access-skbk4\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.764814 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.764789 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.808387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.760882 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.893702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.780335 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.893959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.780583 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.894104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.781261 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9332ef-beec-4b46-9617-36d8ce8812c5-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.894245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.781497 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:28.894385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:28.834280 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:29.002911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:29.002872 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:29.006809 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:29.006734 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:29.057642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:29.032373 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skbk4\" (UniqueName: \"kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-kube-api-access-skbk4\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:29.057642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:29.032905 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:29.217309 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:29.216991 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:29.899017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:29.898962 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt"] Oct 31 13:28:29.909971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:29.909157 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:29.909971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:29.909249 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:29.909971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:29.909299 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:30.396073 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:30.396039 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" event={"ID":"5a9332ef-beec-4b46-9617-36d8ce8812c5","Type":"ContainerStarted","Data":"a443ce7ba02030173423306a865f35938863803713af01ee5e84116b6e294410"} Oct 31 13:28:30.396254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:30.396092 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" event={"ID":"5a9332ef-beec-4b46-9617-36d8ce8812c5","Type":"ContainerStarted","Data":"a2d1fb616db1b64ab6bc274cb4e89d7e27157509f9f56ef40c4013531d7066fb"} Oct 31 13:28:30.424502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:30.424453 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" podStartSLOduration=2.424439093 podStartE2EDuration="2.424439093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:30.421246145 +0000 UTC m=+1478.566989977" watchObservedRunningTime="2026-10-31 13:28:30.424439093 +0000 UTC m=+1478.570182920" Oct 31 13:28:31.218523 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:31.218493 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:31.220117 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:31.220093 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 13:28:31.220252 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:31.220139 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" podUID="5a9332ef-beec-4b46-9617-36d8ce8812c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 13:28:31.797814 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:28:31.797774 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:28:32.218773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:32.218740 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 13:28:32.219156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:32.218791 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" podUID="5a9332ef-beec-4b46-9617-36d8ce8812c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 13:28:33.218147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:33.218115 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 13:28:33.218336 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:33.218164 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" podUID="5a9332ef-beec-4b46-9617-36d8ce8812c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 13:28:34.218336 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:34.218302 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 13:28:34.218762 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:34.218354 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" podUID="5a9332ef-beec-4b46-9617-36d8ce8812c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 13:28:35.218682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.218644 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 13:28:35.219171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.218702 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" podUID="5a9332ef-beec-4b46-9617-36d8ce8812c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 13:28:35.592157 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.592066 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb"] Oct 31 13:28:35.597430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.597406 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.604411 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.604385 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb"] Oct 31 13:28:35.604744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.604728 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7b3ddd\"/\"openshift-service-ca.crt\"" Oct 31 13:28:35.606346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.606301 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7b3ddd\"/\"distinct-2-w0-7b3ddd-kserve-self-signed-certs\"" Oct 31 13:28:35.606441 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.606361 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-7b3ddd\"/\"kube-root-ca.crt\"" Oct 31 13:28:35.606608 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.606573 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-7b3ddd\"/\"default-dockercfg-bqk6t\"" Oct 31 13:28:35.762620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.762478 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-dshm\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.762825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.762735 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-model-cache\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.762825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.762768 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tls-certs\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.762825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.762799 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tmp-dir\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.763010 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.762883 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29wz8\" (UniqueName: \"kubernetes.io/projected/91e86d6a-00ba-4709-b2aa-0c8deda6b802-kube-api-access-29wz8\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.763010 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.762934 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-home\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.864357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.864258 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-dshm\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.864536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.864401 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-model-cache\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.864536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.864428 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tls-certs\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.864536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.864447 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tmp-dir\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.864756 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.864535 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29wz8\" (UniqueName: \"kubernetes.io/projected/91e86d6a-00ba-4709-b2aa-0c8deda6b802-kube-api-access-29wz8\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.864756 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.864586 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-home\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.864869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.864848 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-model-cache\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.865027 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.864986 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-home\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.865257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.865232 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tmp-dir\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.867830 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.867792 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-dshm\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.870172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.870147 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tls-certs\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.882095 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.882066 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29wz8\" (UniqueName: \"kubernetes.io/projected/91e86d6a-00ba-4709-b2aa-0c8deda6b802-kube-api-access-29wz8\") pod \"distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:35.916726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:35.916687 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:36.218261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:36.218230 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 31 13:28:36.218412 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:36.218293 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" podUID="5a9332ef-beec-4b46-9617-36d8ce8812c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 31 13:28:36.243105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:36.243081 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb"] Oct 31 13:28:36.244902 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:28:36.244876 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91e86d6a_00ba_4709_b2aa_0c8deda6b802.slice/crio-d906cc8a1825989d4e9221b6d2969a4b599c9471e136562588b33a77af4b253c WatchSource:0}: Error finding container d906cc8a1825989d4e9221b6d2969a4b599c9471e136562588b33a77af4b253c: Status 404 returned error can't find the container with id d906cc8a1825989d4e9221b6d2969a4b599c9471e136562588b33a77af4b253c Oct 31 13:28:36.436244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:36.436208 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" event={"ID":"91e86d6a-00ba-4709-b2aa-0c8deda6b802","Type":"ContainerStarted","Data":"0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe"} Oct 31 13:28:36.436424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:36.436255 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" event={"ID":"91e86d6a-00ba-4709-b2aa-0c8deda6b802","Type":"ContainerStarted","Data":"d906cc8a1825989d4e9221b6d2969a4b599c9471e136562588b33a77af4b253c"} Oct 31 13:28:36.460295 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:36.460240 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podStartSLOduration=1.460226837 podStartE2EDuration="1.460226837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:36.456374756 +0000 UTC m=+1484.602118586" watchObservedRunningTime="2026-10-31 13:28:36.460226837 +0000 UTC m=+1484.605970710" Oct 31 13:28:36.916882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:36.916841 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:37.474555 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:37.474523 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:37.475069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:37.474991 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:37.476023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:37.476002 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:28:41.766864 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.766832 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8"] Oct 31 13:28:41.771093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.771076 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" Oct 31 13:28:41.778249 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.778223 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:28:41.778379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.778281 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:28:41.781709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.781686 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8"] Oct 31 13:28:41.852376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.852340 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d75nq\" (UniqueName: \"kubernetes.io/projected/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-kube-api-access-d75nq\") pod \"maas-api-revoke-sub-3a172f8a-r2nv8\" (UID: \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" Oct 31 13:28:41.852571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.852548 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a172f8a-r2nv8\" (UID: \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" Oct 31 13:28:41.852667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.852653 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a172f8a-r2nv8\" (UID: \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" Oct 31 13:28:41.954472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.954433 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a172f8a-r2nv8\" (UID: \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" Oct 31 13:28:41.954688 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.954492 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a172f8a-r2nv8\" (UID: \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" Oct 31 13:28:41.954688 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.954648 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d75nq\" (UniqueName: \"kubernetes.io/projected/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-kube-api-access-d75nq\") pod \"maas-api-revoke-sub-3a172f8a-r2nv8\" (UID: \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" Oct 31 13:28:41.955132 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.955107 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a172f8a-r2nv8\" (UID: \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" Oct 31 13:28:41.956911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.956888 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a172f8a-r2nv8\" (UID: \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" Oct 31 13:28:41.968419 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:41.968394 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d75nq\" (UniqueName: \"kubernetes.io/projected/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-kube-api-access-d75nq\") pod \"maas-api-revoke-sub-3a172f8a-r2nv8\" (UID: \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" Oct 31 13:28:42.087747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:42.087678 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" Oct 31 13:28:42.367609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:42.367562 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8"] Oct 31 13:28:42.368219 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:28:42.368193 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0916e8e9_3cae_4e5b_86db_ea508f3d24f4.slice/crio-fccd9788c4da109d42a871ed912cbabe905ce9f796e0bb8782b370869e43b782 WatchSource:0}: Error finding container fccd9788c4da109d42a871ed912cbabe905ce9f796e0bb8782b370869e43b782: Status 404 returned error can't find the container with id fccd9788c4da109d42a871ed912cbabe905ce9f796e0bb8782b370869e43b782 Oct 31 13:28:42.476554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:42.476517 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" event={"ID":"0916e8e9-3cae-4e5b-86db-ea508f3d24f4","Type":"ContainerStarted","Data":"fccd9788c4da109d42a871ed912cbabe905ce9f796e0bb8782b370869e43b782"} Oct 31 13:28:43.485877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.485788 5532 generic.go:356] "Generic (PLEG): container finished" podID="0916e8e9-3cae-4e5b-86db-ea508f3d24f4" containerID="1a886f765485a6be1075519aeba3ff4b3e8164ac95fff22ff3981b8e168e9a55" exitCode=0 Oct 31 13:28:43.486313 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.485883 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" event={"ID":"0916e8e9-3cae-4e5b-86db-ea508f3d24f4","Type":"ContainerDied","Data":"1a886f765485a6be1075519aeba3ff4b3e8164ac95fff22ff3981b8e168e9a55"} Oct 31 13:28:43.660968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.660919 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j"] Oct 31 13:28:43.665266 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.665239 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.686967 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.686936 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j"] Oct 31 13:28:43.770372 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.770285 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj"] Oct 31 13:28:43.781090 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.781035 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:43.784895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.784716 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.784895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.784858 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.785146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.784900 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.785146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.784958 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.785146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.785018 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.785146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.785118 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.785433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.785149 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxnl8\" (UniqueName: \"kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-kube-api-access-qxnl8\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.785433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.785206 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.785433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.785250 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.793559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.793523 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj"] Oct 31 13:28:43.794517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.794489 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c82140\"/\"distinct-2-w3-c82140-kserve-self-signed-certs\"" Oct 31 13:28:43.886776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.886734 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.886965 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.886863 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.886965 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.886902 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.886965 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.886954 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.887162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.886983 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tmp-dir\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:43.887162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.887012 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-home\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:43.887162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.887041 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-model-cache\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:43.887162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.887089 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.887162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.887123 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-dshm\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:43.887425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.887187 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbsr5\" (UniqueName: \"kubernetes.io/projected/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-kube-api-access-mbsr5\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:43.887425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.887374 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.887425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.887413 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxnl8\" (UniqueName: \"kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-kube-api-access-qxnl8\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.887577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.887491 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-certs\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.887577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.887532 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istiod-ca-cert\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.887742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.887727 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-data\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.887844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.887812 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tls-certs\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:43.888134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.888115 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.888711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.888629 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-credential-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.888711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.888697 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.889044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.889023 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-socket\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.889745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.889723 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-envoy\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.890352 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.890331 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-podinfo\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.900015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.899986 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-token\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.900109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.900023 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxnl8\" (UniqueName: \"kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-kube-api-access-qxnl8\") pod \"e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:43.966283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.966237 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t"] Oct 31 13:28:43.972197 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.972173 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:43.986332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.986303 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t"] Oct 31 13:28:43.991067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.991030 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbsr5\" (UniqueName: \"kubernetes.io/projected/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-kube-api-access-mbsr5\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:43.992540 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.992217 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tls-certs\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:43.995273 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:43.992718 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:44.014430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.014265 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tmp-dir\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:44.014430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.014314 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-home\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:44.014430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.014379 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-model-cache\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:44.014759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.014498 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-dshm\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:44.015744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.015264 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-home\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:44.015744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.015656 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tmp-dir\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:44.016960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.015932 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-model-cache\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:44.021743 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.021685 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-dshm\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:44.045056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.044979 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbsr5\" (UniqueName: \"kubernetes.io/projected/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-kube-api-access-mbsr5\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:44.047786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.047725 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tls-certs\") pod \"distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:44.115749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.115706 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.115924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.115794 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.115924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.115838 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v89jw\" (UniqueName: \"kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-kube-api-access-v89jw\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.115924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.115871 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:44.116096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.115879 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.116096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.116041 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.116096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.116082 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.116259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.116182 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.116259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.116220 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.116363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.116341 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.217295 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.217263 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.217433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.217317 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.217433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.217371 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.217433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.217408 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.217625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.217609 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.217705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.217687 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.217766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.217754 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.217820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.217789 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.217880 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.217796 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v89jw\" (UniqueName: \"kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-kube-api-access-v89jw\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.217880 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.217838 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.217974 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.217876 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.218363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.218339 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.219285 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.219237 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.219627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.219554 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.220449 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.220426 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.222509 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.222469 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.231440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.231399 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.231565 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.231544 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v89jw\" (UniqueName: \"kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-kube-api-access-v89jw\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.293044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.292901 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:44.317086 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:28:44.317036 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a204c6_cc2c_40ff_8c62_e5648d4280e1.slice/crio-e25e41f91cb7265dd3b0981f7b52ad28f5c7d703edaef684d090ca0fed34c8c8 WatchSource:0}: Error finding container e25e41f91cb7265dd3b0981f7b52ad28f5c7d703edaef684d090ca0fed34c8c8: Status 404 returned error can't find the container with id e25e41f91cb7265dd3b0981f7b52ad28f5c7d703edaef684d090ca0fed34c8c8 Oct 31 13:28:44.317199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.317144 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j"] Oct 31 13:28:44.319584 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.319550 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:44.319770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.319641 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:44.319770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.319687 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:44.441196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.441164 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj"] Oct 31 13:28:44.446393 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:28:44.446312 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77ce8cc3_c32c_4441_be3c_4ca957f5c7be.slice/crio-f672b3c116a3d79806c66bedcf34c7256a53b04f5901881157c87defa6eff285 WatchSource:0}: Error finding container f672b3c116a3d79806c66bedcf34c7256a53b04f5901881157c87defa6eff285: Status 404 returned error can't find the container with id f672b3c116a3d79806c66bedcf34c7256a53b04f5901881157c87defa6eff285 Oct 31 13:28:44.496084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.496055 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" event={"ID":"92a204c6-cc2c-40ff-8c62-e5648d4280e1","Type":"ContainerStarted","Data":"0f1abfd9ff5a19595d96eb79954e4d0a54033163e37e4f6c69353569ba03cc61"} Oct 31 13:28:44.496458 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.496111 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" event={"ID":"92a204c6-cc2c-40ff-8c62-e5648d4280e1","Type":"ContainerStarted","Data":"e25e41f91cb7265dd3b0981f7b52ad28f5c7d703edaef684d090ca0fed34c8c8"} Oct 31 13:28:44.498338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.498314 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" event={"ID":"77ce8cc3-c32c-4441-be3c-4ca957f5c7be","Type":"ContainerStarted","Data":"f672b3c116a3d79806c66bedcf34c7256a53b04f5901881157c87defa6eff285"} Oct 31 13:28:44.534465 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.533991 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" podStartSLOduration=1.533964443 podStartE2EDuration="1.533964443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:44.532005105 +0000 UTC m=+1492.677748939" watchObservedRunningTime="2026-10-31 13:28:44.533964443 +0000 UTC m=+1492.679708272" Oct 31 13:28:44.593397 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.590117 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t"] Oct 31 13:28:44.598119 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:28:44.598080 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a27b459_9ad5_4ade_87f3_6ef3ef735e30.slice/crio-e6be2a43d63cd1bf93e77bed74229782c6d3a348e2f619c1d371fc02304240f8 WatchSource:0}: Error finding container e6be2a43d63cd1bf93e77bed74229782c6d3a348e2f619c1d371fc02304240f8: Status 404 returned error can't find the container with id e6be2a43d63cd1bf93e77bed74229782c6d3a348e2f619c1d371fc02304240f8 Oct 31 13:28:44.608691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.608370 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:44.608691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.608469 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:44.608691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.608513 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:44.685155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.685132 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" Oct 31 13:28:44.838061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.837974 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-kube-api-access-d75nq\" (UniqueName: \"kubernetes.io/projected/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-kube-api-access-d75nq\") pod \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\" (UID: \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\") " Oct 31 13:28:44.838061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.838030 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-xks-ca\") pod \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\" (UID: \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\") " Oct 31 13:28:44.838061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.838055 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-openshift-ca\") pod \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\" (UID: \"0916e8e9-3cae-4e5b-86db-ea508f3d24f4\") " Oct 31 13:28:44.838511 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.838483 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-openshift-ca" pod "0916e8e9-3cae-4e5b-86db-ea508f3d24f4" (UID: "0916e8e9-3cae-4e5b-86db-ea508f3d24f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:44.838727 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.838711 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:44.840224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.840204 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-kube-api-access-d75nq" pod "0916e8e9-3cae-4e5b-86db-ea508f3d24f4" (UID: "0916e8e9-3cae-4e5b-86db-ea508f3d24f4"). InnerVolumeSpecName "kube-api-access-d75nq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:44.841289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.841259 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-xks-ca" pod "0916e8e9-3cae-4e5b-86db-ea508f3d24f4" (UID: "0916e8e9-3cae-4e5b-86db-ea508f3d24f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:28:44.943339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.940448 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d75nq\" (UniqueName: \"kubernetes.io/projected/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-kube-api-access-d75nq\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:44.943339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.940487 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0916e8e9-3cae-4e5b-86db-ea508f3d24f4-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:44.996220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:44.996183 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:45.511120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:45.511077 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" event={"ID":"0916e8e9-3cae-4e5b-86db-ea508f3d24f4","Type":"ContainerDied","Data":"fccd9788c4da109d42a871ed912cbabe905ce9f796e0bb8782b370869e43b782"} Oct 31 13:28:45.511702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:45.511139 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fccd9788c4da109d42a871ed912cbabe905ce9f796e0bb8782b370869e43b782" Oct 31 13:28:45.511702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:45.511090 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8" Oct 31 13:28:45.514884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:45.514575 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" event={"ID":"77ce8cc3-c32c-4441-be3c-4ca957f5c7be","Type":"ContainerStarted","Data":"3dbe0f5ca021aa62c51d3ac34c28c4892636d8c2aa386ecef9a8ff9425b70615"} Oct 31 13:28:45.518675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:45.518650 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" event={"ID":"0a27b459-9ad5-4ade-87f3-6ef3ef735e30","Type":"ContainerStarted","Data":"1ac4a737aaae7c4b67f8dd0ec04ea5c623b623e7db8fbe9f836bfbdd80c25487"} Oct 31 13:28:45.518793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:45.518689 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" event={"ID":"0a27b459-9ad5-4ade-87f3-6ef3ef735e30","Type":"ContainerStarted","Data":"e6be2a43d63cd1bf93e77bed74229782c6d3a348e2f619c1d371fc02304240f8"} Oct 31 13:28:45.538178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:45.538132 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podStartSLOduration=2.538119756 podStartE2EDuration="2.538119756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:45.536706041 +0000 UTC m=+1493.682449866" watchObservedRunningTime="2026-10-31 13:28:45.538119756 +0000 UTC m=+1493.683863584" Oct 31 13:28:45.601651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:45.601511 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" podStartSLOduration=2.601490419 podStartE2EDuration="2.601490419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:45.599967499 +0000 UTC m=+1493.745711329" watchObservedRunningTime="2026-10-31 13:28:45.601490419 +0000 UTC m=+1493.747234247" Oct 31 13:28:45.917660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:45.917624 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:45.930701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:45.930675 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:45.997538 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:45.997508 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 13:28:45.997712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:45.997561 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" podUID="92a204c6-cc2c-40ff-8c62-e5648d4280e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 13:28:46.117382 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:46.117011 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:46.293562 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:46.293463 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:46.294536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:46.294512 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 13:28:46.294590 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:46.294565 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" podUID="0a27b459-9ad5-4ade-87f3-6ef3ef735e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 13:28:46.536116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:46.536088 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:28:46.996156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:46.996119 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 13:28:46.996315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:46.996172 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" podUID="92a204c6-cc2c-40ff-8c62-e5648d4280e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 13:28:47.294686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:47.294585 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 13:28:47.294686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:47.294652 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" podUID="0a27b459-9ad5-4ade-87f3-6ef3ef735e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 13:28:47.996300 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:47.996257 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 13:28:47.996736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:47.996324 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" podUID="92a204c6-cc2c-40ff-8c62-e5648d4280e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 13:28:48.293676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:48.293327 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 13:28:48.293676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:48.293384 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" podUID="0a27b459-9ad5-4ade-87f3-6ef3ef735e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 13:28:48.996404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:48.996359 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 13:28:48.996868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:48.996425 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" podUID="92a204c6-cc2c-40ff-8c62-e5648d4280e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 13:28:49.128872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.128839 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782"] Oct 31 13:28:49.129468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.129453 5532 cpu_manager.go:414] "containerMap: removing container" podUID="0916e8e9-3cae-4e5b-86db-ea508f3d24f4" containerName="revoke-keys" Oct 31 13:28:49.129518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.129469 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0916e8e9-3cae-4e5b-86db-ea508f3d24f4" containerName="revoke-keys" Oct 31 13:28:49.129579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.129569 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="0916e8e9-3cae-4e5b-86db-ea508f3d24f4" containerName="revoke-keys" Oct 31 13:28:49.134517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.134497 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.165618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.151549 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.165618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.151635 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.165618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.151734 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.165618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.151780 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.165618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.151893 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8xzq\" (UniqueName: \"kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-kube-api-access-g8xzq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.165618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.151954 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a503272-9747-4a45-abdb-62cdbb0a2560-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.165618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.151996 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.191619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.181694 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 13:28:49.191619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.182109 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 13:28:49.191619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.182895 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 13:28:49.191619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.182995 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zfm9b\"" Oct 31 13:28:49.212345 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.212091 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782"] Oct 31 13:28:49.293767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.293674 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8xzq\" (UniqueName: \"kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-kube-api-access-g8xzq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.294873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.294847 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a503272-9747-4a45-abdb-62cdbb0a2560-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.295462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.295443 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.296151 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.296134 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.296687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.296670 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.297144 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.297127 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.298044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.298024 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.298445 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.297951 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.298526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.294745 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 13:28:49.298526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.298503 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" podUID="0a27b459-9ad5-4ade-87f3-6ef3ef735e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 13:28:49.298711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.298692 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.299273 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.299254 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a503272-9747-4a45-abdb-62cdbb0a2560-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.301487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.301418 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.303699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.303681 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.308011 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.307989 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8xzq\" (UniqueName: \"kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-kube-api-access-g8xzq\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.309901 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.309879 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-bb782\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:49.522308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:49.522262 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:50.000236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:50.000198 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 13:28:50.000760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:50.000258 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" podUID="92a204c6-cc2c-40ff-8c62-e5648d4280e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 13:28:50.036241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:50.036209 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782"] Oct 31 13:28:50.293557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:50.293452 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 13:28:50.293557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:50.293537 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" podUID="0a27b459-9ad5-4ade-87f3-6ef3ef735e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 13:28:50.625665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:50.625533 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" event={"ID":"0a503272-9747-4a45-abdb-62cdbb0a2560","Type":"ContainerStarted","Data":"a995c9dd9fda9749ae79e3c5862bc3071b124ea2a069c30ba11e87c270d5f96a"} Oct 31 13:28:50.995942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:50.995888 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 31 13:28:50.996165 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:50.995954 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" podUID="92a204c6-cc2c-40ff-8c62-e5648d4280e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 31 13:28:51.293505 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:51.293420 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 31 13:28:51.293505 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:51.293468 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" podUID="0a27b459-9ad5-4ade-87f3-6ef3ef735e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 31 13:28:52.010064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.010032 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:52.010397 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.010376 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:52.011409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.011356 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:28:52.298938 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.298858 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:52.300009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.299983 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:52.303656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.303630 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:28:52.924512 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.924476 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc"] Oct 31 13:28:52.931875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.931846 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" Oct 31 13:28:52.939124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.939091 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:28:52.939124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.939096 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:28:52.942385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.942361 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc"] Oct 31 13:28:52.977476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.977442 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b81d889-d246-4041-8380-4974723b2689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40efb871-jl4zc\" (UID: \"7b81d889-d246-4041-8380-4974723b2689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" Oct 31 13:28:52.977715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.977626 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrrm5\" (UniqueName: \"kubernetes.io/projected/7b81d889-d246-4041-8380-4974723b2689-kube-api-access-xrrm5\") pod \"maas-api-revoke-sub-40efb871-jl4zc\" (UID: \"7b81d889-d246-4041-8380-4974723b2689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" Oct 31 13:28:52.977715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:52.977666 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b81d889-d246-4041-8380-4974723b2689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40efb871-jl4zc\" (UID: \"7b81d889-d246-4041-8380-4974723b2689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" Oct 31 13:28:53.078502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:53.078461 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrrm5\" (UniqueName: \"kubernetes.io/projected/7b81d889-d246-4041-8380-4974723b2689-kube-api-access-xrrm5\") pod \"maas-api-revoke-sub-40efb871-jl4zc\" (UID: \"7b81d889-d246-4041-8380-4974723b2689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" Oct 31 13:28:53.078502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:53.078508 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b81d889-d246-4041-8380-4974723b2689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40efb871-jl4zc\" (UID: \"7b81d889-d246-4041-8380-4974723b2689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" Oct 31 13:28:53.078735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:53.078680 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b81d889-d246-4041-8380-4974723b2689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40efb871-jl4zc\" (UID: \"7b81d889-d246-4041-8380-4974723b2689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" Oct 31 13:28:53.079215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:53.079191 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b81d889-d246-4041-8380-4974723b2689-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40efb871-jl4zc\" (UID: \"7b81d889-d246-4041-8380-4974723b2689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" Oct 31 13:28:53.081172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:53.081149 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b81d889-d246-4041-8380-4974723b2689-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40efb871-jl4zc\" (UID: \"7b81d889-d246-4041-8380-4974723b2689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" Oct 31 13:28:53.094813 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:53.094789 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrrm5\" (UniqueName: \"kubernetes.io/projected/7b81d889-d246-4041-8380-4974723b2689-kube-api-access-xrrm5\") pod \"maas-api-revoke-sub-40efb871-jl4zc\" (UID: \"7b81d889-d246-4041-8380-4974723b2689\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" Oct 31 13:28:53.253326 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:53.253239 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" Oct 31 13:28:53.550455 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:28:53.550410 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b81d889_d246_4041_8380_4974723b2689.slice/crio-0c9249515d07e97b85b4d76a400f13c736722444cff47dfbdb75b81f9132ef28 WatchSource:0}: Error finding container 0c9249515d07e97b85b4d76a400f13c736722444cff47dfbdb75b81f9132ef28: Status 404 returned error can't find the container with id 0c9249515d07e97b85b4d76a400f13c736722444cff47dfbdb75b81f9132ef28 Oct 31 13:28:53.550965 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:53.550939 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc"] Oct 31 13:28:53.653915 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:53.653884 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" event={"ID":"7b81d889-d246-4041-8380-4974723b2689","Type":"ContainerStarted","Data":"0c9249515d07e97b85b4d76a400f13c736722444cff47dfbdb75b81f9132ef28"} Oct 31 13:28:53.726477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:53.726441 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:53.726611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:53.726518 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:28:54.116153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:54.116116 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:54.133717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:54.133676 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:54.663438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:54.663396 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" event={"ID":"0a503272-9747-4a45-abdb-62cdbb0a2560","Type":"ContainerStarted","Data":"8517e6c17953f75a28780bb81f5c23f37842b80fce97141470a51045e7f1b14f"} Oct 31 13:28:54.663937 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:54.663465 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:54.665203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:54.665177 5532 generic.go:356] "Generic (PLEG): container finished" podID="7b81d889-d246-4041-8380-4974723b2689" containerID="68dc0210f892b713cb3a14c7f6da1d02b53a5684047822a51e4b2453e6675785" exitCode=0 Oct 31 13:28:54.665317 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:54.665277 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" event={"ID":"7b81d889-d246-4041-8380-4974723b2689","Type":"ContainerDied","Data":"68dc0210f892b713cb3a14c7f6da1d02b53a5684047822a51e4b2453e6675785"} Oct 31 13:28:54.666007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:54.665986 5532 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-bb782 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:28:54.666118 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:54.666041 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" podUID="0a503272-9747-4a45-abdb-62cdbb0a2560" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:28:54.681379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:54.681351 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:28:54.693727 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:54.693684 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" podStartSLOduration=2.06322472 podStartE2EDuration="5.69367116s" totalImagesPullingTime="3.63044644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:28:54.68927888 +0000 UTC m=+1502.835022708" watchObservedRunningTime="2026-10-31 13:28:54.69367116 +0000 UTC m=+1502.839414987" Oct 31 13:28:55.675533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:55.674892 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:28:55.863271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:55.863246 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" Oct 31 13:28:55.931074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:55.930975 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b81d889-d246-4041-8380-4974723b2689-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b81d889-d246-4041-8380-4974723b2689-maas-api-xks-ca\") pod \"7b81d889-d246-4041-8380-4974723b2689\" (UID: \"7b81d889-d246-4041-8380-4974723b2689\") " Oct 31 13:28:55.931236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:55.931174 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b81d889-d246-4041-8380-4974723b2689-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b81d889-d246-4041-8380-4974723b2689-maas-api-openshift-ca\") pod \"7b81d889-d246-4041-8380-4974723b2689\" (UID: \"7b81d889-d246-4041-8380-4974723b2689\") " Oct 31 13:28:55.931315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:55.931252 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b81d889-d246-4041-8380-4974723b2689-kube-api-access-xrrm5\" (UniqueName: \"kubernetes.io/projected/7b81d889-d246-4041-8380-4974723b2689-kube-api-access-xrrm5\") pod \"7b81d889-d246-4041-8380-4974723b2689\" (UID: \"7b81d889-d246-4041-8380-4974723b2689\") " Oct 31 13:28:55.931638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:55.931583 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b81d889-d246-4041-8380-4974723b2689-maas-api-openshift-ca" pod "7b81d889-d246-4041-8380-4974723b2689" (UID: "7b81d889-d246-4041-8380-4974723b2689"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:28:55.931984 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:55.931945 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b81d889-d246-4041-8380-4974723b2689-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:55.933680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:55.933653 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b81d889-d246-4041-8380-4974723b2689-maas-api-xks-ca" pod "7b81d889-d246-4041-8380-4974723b2689" (UID: "7b81d889-d246-4041-8380-4974723b2689"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:28:55.934041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:55.934018 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b81d889-d246-4041-8380-4974723b2689-kube-api-access-xrrm5" pod "7b81d889-d246-4041-8380-4974723b2689" (UID: "7b81d889-d246-4041-8380-4974723b2689"). InnerVolumeSpecName "kube-api-access-xrrm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:28:56.033733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:56.033705 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b81d889-d246-4041-8380-4974723b2689-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:56.033733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:56.033731 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrrm5\" (UniqueName: \"kubernetes.io/projected/7b81d889-d246-4041-8380-4974723b2689-kube-api-access-xrrm5\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:28:56.683105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:56.683077 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" Oct 31 13:28:56.683105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:56.683105 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc" event={"ID":"7b81d889-d246-4041-8380-4974723b2689","Type":"ContainerDied","Data":"0c9249515d07e97b85b4d76a400f13c736722444cff47dfbdb75b81f9132ef28"} Oct 31 13:28:56.683657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:56.683154 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c9249515d07e97b85b4d76a400f13c736722444cff47dfbdb75b81f9132ef28" Oct 31 13:28:59.207618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:59.206706 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:28:59.243733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:59.243685 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:28:59.340831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:59.340768 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:28:59.366635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:59.354686 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:28:59.400053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:59.385534 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:28:59.434842 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:59.434787 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:28:59.492671 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:59.487847 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:28:59.551761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:59.551710 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:28:59.580792 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:59.580737 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:28:59.671786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:59.671731 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:28:59.843580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:59.843475 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:28:59.928932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:28:59.928881 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:29:00.228618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.226820 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:00.259715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.255692 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:00.405584 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.387247 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:29:00.405584 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.389423 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:00.446654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.434857 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:29:00.446654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.434930 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:29:00.446654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.434970 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:29:00.446654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.435006 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:29:00.465895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.447125 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:29:00.465895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.450114 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:29:00.500837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.494680 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:29:00.500837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.495443 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:29:00.592618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.588744 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:00.592618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.590728 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:29:00.592618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.592160 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:00.592618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.592495 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:29:00.681984 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.672688 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:00.717620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.692715 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:29:00.949551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.949509 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:00.971641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.950381 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:29:00.973509 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:00.961502 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:01.241516 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:01.241415 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:01.262122 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:01.262077 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:01.384894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:01.381294 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:01.504709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:01.398011 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:29:01.504709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:01.399150 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:29:01.504709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:01.456368 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:01.545404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:01.516801 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:01.592831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:01.592781 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:01.602725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:01.602681 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:01.710640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:01.709048 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:01.776767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:01.751106 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" podUID="51b75119-a5cd-4e8c-bd06-3725a6bb30a5" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.29:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:01.960519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:01.960475 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:02.139162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:02.139079 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:29:02.143093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:02.143068 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:29:02.229327 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:02.229293 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:29:02.340279 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:29:02.340237 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:29:02.341993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:02.341913 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:29:02.342399 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:02.342381 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:29:02.346683 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:02.346658 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:29:02.351123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:02.351100 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:29:02.351980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:02.351958 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:29:02.354125 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:02.354105 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:29:02.354312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:02.354297 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:29:06.273033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.273001 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5"] Oct 31 13:29:06.273684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.273667 5532 cpu_manager.go:414] "containerMap: removing container" podUID="7b81d889-d246-4041-8380-4974723b2689" containerName="revoke-keys" Oct 31 13:29:06.273759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.273684 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b81d889-d246-4041-8380-4974723b2689" containerName="revoke-keys" Oct 31 13:29:06.273806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.273796 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b81d889-d246-4041-8380-4974723b2689" containerName="revoke-keys" Oct 31 13:29:06.280474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.280439 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" Oct 31 13:29:06.287034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.287006 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:29:06.287211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.287182 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:29:06.291416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.291391 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5"] Oct 31 13:29:06.408176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.408121 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e4e5df1-s8wt5\" (UID: \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" Oct 31 13:29:06.408754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.408706 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znrm5\" (UniqueName: \"kubernetes.io/projected/e4d9b75a-ffdd-452e-a13f-acb001009bf4-kube-api-access-znrm5\") pod \"maas-api-revoke-sub-1e4e5df1-s8wt5\" (UID: \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" Oct 31 13:29:06.409012 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.408993 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e4e5df1-s8wt5\" (UID: \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" Oct 31 13:29:06.510872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.510834 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e4e5df1-s8wt5\" (UID: \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" Oct 31 13:29:06.511049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.511025 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrm5\" (UniqueName: \"kubernetes.io/projected/e4d9b75a-ffdd-452e-a13f-acb001009bf4-kube-api-access-znrm5\") pod \"maas-api-revoke-sub-1e4e5df1-s8wt5\" (UID: \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" Oct 31 13:29:06.511210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.511193 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e4e5df1-s8wt5\" (UID: \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" Oct 31 13:29:06.511518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.511498 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e4e5df1-s8wt5\" (UID: \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" Oct 31 13:29:06.513524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.513502 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e4e5df1-s8wt5\" (UID: \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" Oct 31 13:29:06.524248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.524190 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znrm5\" (UniqueName: \"kubernetes.io/projected/e4d9b75a-ffdd-452e-a13f-acb001009bf4-kube-api-access-znrm5\") pod \"maas-api-revoke-sub-1e4e5df1-s8wt5\" (UID: \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" Oct 31 13:29:06.605850 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.605822 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" Oct 31 13:29:06.890951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:06.890918 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5"] Oct 31 13:29:06.891626 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:29:06.891571 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d9b75a_ffdd_452e_a13f_acb001009bf4.slice/crio-019f8476e110d2b7da2445b9430d92e0075ec59efabed284e9814a7e713c5595 WatchSource:0}: Error finding container 019f8476e110d2b7da2445b9430d92e0075ec59efabed284e9814a7e713c5595: Status 404 returned error can't find the container with id 019f8476e110d2b7da2445b9430d92e0075ec59efabed284e9814a7e713c5595 Oct 31 13:29:07.093662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:07.093630 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" event={"ID":"e4d9b75a-ffdd-452e-a13f-acb001009bf4","Type":"ContainerStarted","Data":"019f8476e110d2b7da2445b9430d92e0075ec59efabed284e9814a7e713c5595"} Oct 31 13:29:08.103666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:08.103560 5532 generic.go:356] "Generic (PLEG): container finished" podID="e4d9b75a-ffdd-452e-a13f-acb001009bf4" containerID="a4e955e7acb7a86600bc3c64888d0f816aa3c6f8bbb9a23346fed17f6e57fb88" exitCode=0 Oct 31 13:29:08.103666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:08.103644 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" event={"ID":"e4d9b75a-ffdd-452e-a13f-acb001009bf4","Type":"ContainerDied","Data":"a4e955e7acb7a86600bc3c64888d0f816aa3c6f8bbb9a23346fed17f6e57fb88"} Oct 31 13:29:09.287166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:09.286848 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" Oct 31 13:29:09.350294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:09.350261 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-openshift-ca\") pod \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\" (UID: \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\") " Oct 31 13:29:09.350503 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:09.350399 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d9b75a-ffdd-452e-a13f-acb001009bf4-kube-api-access-znrm5\" (UniqueName: \"kubernetes.io/projected/e4d9b75a-ffdd-452e-a13f-acb001009bf4-kube-api-access-znrm5\") pod \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\" (UID: \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\") " Oct 31 13:29:09.350635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:09.350579 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-xks-ca\") pod \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\" (UID: \"e4d9b75a-ffdd-452e-a13f-acb001009bf4\") " Oct 31 13:29:09.351754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:09.351722 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-openshift-ca" pod "e4d9b75a-ffdd-452e-a13f-acb001009bf4" (UID: "e4d9b75a-ffdd-452e-a13f-acb001009bf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:09.361694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:09.354322 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d9b75a-ffdd-452e-a13f-acb001009bf4-kube-api-access-znrm5" pod "e4d9b75a-ffdd-452e-a13f-acb001009bf4" (UID: "e4d9b75a-ffdd-452e-a13f-acb001009bf4"). InnerVolumeSpecName "kube-api-access-znrm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:09.361694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:09.354404 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-xks-ca" pod "e4d9b75a-ffdd-452e-a13f-acb001009bf4" (UID: "e4d9b75a-ffdd-452e-a13f-acb001009bf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:29:09.453036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:09.453005 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:09.453036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:09.453036 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4d9b75a-ffdd-452e-a13f-acb001009bf4-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:09.453241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:09.453047 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znrm5\" (UniqueName: \"kubernetes.io/projected/e4d9b75a-ffdd-452e-a13f-acb001009bf4-kube-api-access-znrm5\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:10.119887 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:10.119860 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" Oct 31 13:29:10.120064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:10.119890 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5" event={"ID":"e4d9b75a-ffdd-452e-a13f-acb001009bf4","Type":"ContainerDied","Data":"019f8476e110d2b7da2445b9430d92e0075ec59efabed284e9814a7e713c5595"} Oct 31 13:29:10.120064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:10.119948 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="019f8476e110d2b7da2445b9430d92e0075ec59efabed284e9814a7e713c5595" Oct 31 13:29:13.363390 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.363358 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8"] Oct 31 13:29:13.364176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.364153 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e4d9b75a-ffdd-452e-a13f-acb001009bf4" containerName="revoke-keys" Oct 31 13:29:13.364176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.364176 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d9b75a-ffdd-452e-a13f-acb001009bf4" containerName="revoke-keys" Oct 31 13:29:13.364436 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.364421 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d9b75a-ffdd-452e-a13f-acb001009bf4" containerName="revoke-keys" Oct 31 13:29:13.368057 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.368036 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" Oct 31 13:29:13.376848 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.376479 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:29:13.376848 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.376637 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:29:13.380527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.380501 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8"] Oct 31 13:29:13.513903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.513870 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ec2677-nb6b8\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" Oct 31 13:29:13.514077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.513928 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ec2677-nb6b8\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" Oct 31 13:29:13.514124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.514072 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq9q7\" (UniqueName: \"kubernetes.io/projected/f93043d3-9e55-4d63-85e7-47e7e04076c0-kube-api-access-bq9q7\") pod \"maas-api-revoke-sub-e1ec2677-nb6b8\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" Oct 31 13:29:13.601015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.600982 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d"] Oct 31 13:29:13.605537 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.605511 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.617574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.617087 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq9q7\" (UniqueName: \"kubernetes.io/projected/f93043d3-9e55-4d63-85e7-47e7e04076c0-kube-api-access-bq9q7\") pod \"maas-api-revoke-sub-e1ec2677-nb6b8\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" Oct 31 13:29:13.624366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.623156 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ec2677-nb6b8\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" Oct 31 13:29:13.624492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.624424 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ec2677-nb6b8\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" Oct 31 13:29:13.625410 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.625355 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1ec2677-nb6b8\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" Oct 31 13:29:13.629487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.629456 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d"] Oct 31 13:29:13.629645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.629558 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1ec2677-nb6b8\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" Oct 31 13:29:13.638142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.638110 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq9q7\" (UniqueName: \"kubernetes.io/projected/f93043d3-9e55-4d63-85e7-47e7e04076c0-kube-api-access-bq9q7\") pod \"maas-api-revoke-sub-e1ec2677-nb6b8\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" Oct 31 13:29:13.686712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.686677 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" Oct 31 13:29:13.726621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.726573 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.726806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.726719 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.726806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.726745 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.726806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.726793 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.726988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.726815 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.726988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.726885 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.726988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.726918 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.726988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.726937 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w225v\" (UniqueName: \"kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-kube-api-access-w225v\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.727222 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.727084 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.828571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.828529 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.828764 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.828607 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.828764 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.828711 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.828902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.828763 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.828902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.828796 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w225v\" (UniqueName: \"kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-kube-api-access-w225v\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.829021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.828905 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.829021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.828959 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.829124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.829032 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-certs\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.829124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.829061 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.829507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.829478 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.829670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.829652 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-data\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.829985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.829930 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.830096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.830071 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-credential-socket\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.830329 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.830305 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istiod-ca-cert\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.832371 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.832345 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-envoy\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.832489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.832382 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-podinfo\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.842705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.842676 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-token\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.842925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.842900 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w225v\" (UniqueName: \"kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-kube-api-access-w225v\") pod \"e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.933923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.933872 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:13.959307 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.959262 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk"] Oct 31 13:29:13.973484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.973295 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd"] Oct 31 13:29:13.974428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.973738 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:13.981849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.978916 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk"] Oct 31 13:29:13.981849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.979049 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:13.981849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:13.980029 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd"] Oct 31 13:29:14.031650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.027754 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr"] Oct 31 13:29:14.037395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.036871 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8"] Oct 31 13:29:14.037395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.037035 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.037395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.037102 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr"] Oct 31 13:29:14.140676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.140645 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.140826 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.140690 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.140826 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.140771 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.140905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.140875 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.140947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.140933 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bpdg\" (UniqueName: \"kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-kube-api-access-9bpdg\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.140979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.140960 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.141024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.140981 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.141092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141067 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/225e725e-7d21-4878-ac58-f9b9bb55f144-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.141226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141195 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.141358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141238 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.141358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141259 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde88896-6051-4d05-8392-8b7eec0b2ccf-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.141358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141303 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.141358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141322 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.141358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141342 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.141630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141387 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.141630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141439 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.141630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141475 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f61058-6f46-454c-a7a6-fea17390fc1c-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.141630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141507 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.141630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141524 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.141630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141544 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.141947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141693 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.141947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141913 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.141947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141944 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.142109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.141974 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.142109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.142002 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2nw5\" (UniqueName: \"kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-kube-api-access-z2nw5\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.142109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.142016 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.142109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.142036 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbj2n\" (UniqueName: \"kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-kube-api-access-hbj2n\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.150921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.150896 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" event={"ID":"f93043d3-9e55-4d63-85e7-47e7e04076c0","Type":"ContainerStarted","Data":"bc2aab2e822f18c9c42b4a7b153f9c978012a3ca7c3ba511c53f47139bbc3f98"} Oct 31 13:29:14.243213 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243136 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.243213 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243186 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.243430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243219 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.243430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243280 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.243430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243359 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.243430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243413 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f61058-6f46-454c-a7a6-fea17390fc1c-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.243580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243465 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.243580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243491 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.243580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243521 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.243580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243550 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.243580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243570 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.243831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243588 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.243831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243646 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.243831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243676 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2nw5\" (UniqueName: \"kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-kube-api-access-z2nw5\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.243831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243677 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.243831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243673 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.243831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243742 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.243831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243783 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbj2n\" (UniqueName: \"kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-kube-api-access-hbj2n\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.243831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243828 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.244235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243878 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.244235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243913 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.244235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243960 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.244235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.243982 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.244235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244027 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bpdg\" (UniqueName: \"kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-kube-api-access-9bpdg\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.244235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244055 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.244235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244081 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.244235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244110 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/225e725e-7d21-4878-ac58-f9b9bb55f144-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.244235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244162 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.245691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.245637 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/225e725e-7d21-4878-ac58-f9b9bb55f144-istiod-ca-cert\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.245691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244167 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.245888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.245746 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.245888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.245803 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde88896-6051-4d05-8392-8b7eec0b2ccf-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.246003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244733 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-credential-socket\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.246120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244872 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-credential-socket\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.246120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244966 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.246120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244988 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-data\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.246120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.245009 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.246120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244585 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f61058-6f46-454c-a7a6-fea17390fc1c-istiod-ca-cert\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.246120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244648 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-data\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.246439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244702 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-certs\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.246439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.246418 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.246680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.244424 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-certs\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.246959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.246920 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-envoy\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.247101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.247082 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde88896-6051-4d05-8392-8b7eec0b2ccf-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.247171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.247125 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-podinfo\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.247882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.247861 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-podinfo\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.248449 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.248427 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-envoy\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.248544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.248449 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.257250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.257227 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbj2n\" (UniqueName: \"kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-kube-api-access-hbj2n\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.257348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.257321 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-token\") pod \"e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.257750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.257730 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-token\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.258037 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.258011 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2nw5\" (UniqueName: \"kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-kube-api-access-z2nw5\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.258573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.258553 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bpdg\" (UniqueName: \"kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-kube-api-access-9bpdg\") pod \"e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.258758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.258738 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.271029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.271007 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d"] Oct 31 13:29:14.271399 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:29:14.271373 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b1ba1d2_444c_4dfc_adea_6458e9e4c590.slice/crio-3a193ac8f5ee80c75b89b62b5fda48ce21d0667bc0c14374c20d431fc9551a3e WatchSource:0}: Error finding container 3a193ac8f5ee80c75b89b62b5fda48ce21d0667bc0c14374c20d431fc9551a3e: Status 404 returned error can't find the container with id 3a193ac8f5ee80c75b89b62b5fda48ce21d0667bc0c14374c20d431fc9551a3e Oct 31 13:29:14.274123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.274091 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:14.274208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.274172 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:14.274285 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.274220 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:14.303354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.303332 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:14.325469 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.325445 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:14.373224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.373192 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:14.630375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.630345 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk"] Oct 31 13:29:14.634307 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.634274 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:14.634406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.634337 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:14.634406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.634366 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:14.666799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.666769 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd"] Oct 31 13:29:14.668764 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:29:14.668730 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f61058_6f46_454c_a7a6_fea17390fc1c.slice/crio-9e2ae0b04836137bb8e5b3f9fbd40d89d6e5fb6498491641140de3a5131d695f WatchSource:0}: Error finding container 9e2ae0b04836137bb8e5b3f9fbd40d89d6e5fb6498491641140de3a5131d695f: Status 404 returned error can't find the container with id 9e2ae0b04836137bb8e5b3f9fbd40d89d6e5fb6498491641140de3a5131d695f Oct 31 13:29:14.671253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.671223 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:14.671339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.671286 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:14.671339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.671315 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:14.738648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.738623 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr"] Oct 31 13:29:14.750128 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.750092 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:14.750265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.750178 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:14.750265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:14.750235 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:15.185615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.185324 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" event={"ID":"2b1ba1d2-444c-4dfc-adea-6458e9e4c590","Type":"ContainerStarted","Data":"7bde2f98e7539ec838be01a01dfb79dbe886ddd83aca39c3fe45cfa77aedfdd0"} Oct 31 13:29:15.185615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.185407 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" event={"ID":"2b1ba1d2-444c-4dfc-adea-6458e9e4c590","Type":"ContainerStarted","Data":"3a193ac8f5ee80c75b89b62b5fda48ce21d0667bc0c14374c20d431fc9551a3e"} Oct 31 13:29:15.191439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.188183 5532 generic.go:356] "Generic (PLEG): container finished" podID="f93043d3-9e55-4d63-85e7-47e7e04076c0" containerID="652d863d57a8df9596029ce32c67be6a9ed6d883f4e5fb0e6765db00a6a6e485" exitCode=0 Oct 31 13:29:15.191439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.188307 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" event={"ID":"f93043d3-9e55-4d63-85e7-47e7e04076c0","Type":"ContainerDied","Data":"652d863d57a8df9596029ce32c67be6a9ed6d883f4e5fb0e6765db00a6a6e485"} Oct 31 13:29:15.191439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.190324 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" event={"ID":"a3f61058-6f46-454c-a7a6-fea17390fc1c","Type":"ContainerStarted","Data":"8754d56bb37d4beace8f260f1de071b9bcba31a2de693b0fe4fce6c426dd261b"} Oct 31 13:29:15.191439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.190366 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" event={"ID":"a3f61058-6f46-454c-a7a6-fea17390fc1c","Type":"ContainerStarted","Data":"9e2ae0b04836137bb8e5b3f9fbd40d89d6e5fb6498491641140de3a5131d695f"} Oct 31 13:29:15.195818 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.195796 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" event={"ID":"225e725e-7d21-4878-ac58-f9b9bb55f144","Type":"ContainerStarted","Data":"45c723888cc9f8b5d600222bdaa5b1d81c1b30bdd8bf02fe07b6f4c12158d01b"} Oct 31 13:29:15.195961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.195948 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" event={"ID":"225e725e-7d21-4878-ac58-f9b9bb55f144","Type":"ContainerStarted","Data":"a73ce05f231d794f7b7346a0db44f90917842608b6ff859e6b7e33e1f671e627"} Oct 31 13:29:15.198285 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.198265 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" event={"ID":"cde88896-6051-4d05-8392-8b7eec0b2ccf","Type":"ContainerStarted","Data":"c98a610e457f1f18f69413843b5176dd9a5107e8d6c6c71818eb17e902fe162a"} Oct 31 13:29:15.198943 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.198926 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" event={"ID":"cde88896-6051-4d05-8392-8b7eec0b2ccf","Type":"ContainerStarted","Data":"3131c7f90cbff98dd42bd45065dc44c690d4e54ad9a2c1a4bb3694e594911767"} Oct 31 13:29:15.286273 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.285849 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" podStartSLOduration=2.285829938 podStartE2EDuration="2.285829938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.218804138 +0000 UTC m=+1523.364547980" watchObservedRunningTime="2026-10-31 13:29:15.285829938 +0000 UTC m=+1523.431573768" Oct 31 13:29:15.287063 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.287014 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" podStartSLOduration=2.287001797 podStartE2EDuration="2.287001797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.261280509 +0000 UTC m=+1523.407024335" watchObservedRunningTime="2026-10-31 13:29:15.287001797 +0000 UTC m=+1523.432745630" Oct 31 13:29:15.304122 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.304052 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:15.305388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.305363 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 13:29:15.305549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.305531 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 13:29:15.325709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.325638 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:15.328609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.328568 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 13:29:15.328791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.328653 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 13:29:15.388291 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.388221 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" podStartSLOduration=2.388194221 podStartE2EDuration="2.388194221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.316532301 +0000 UTC m=+1523.462276130" watchObservedRunningTime="2026-10-31 13:29:15.388194221 +0000 UTC m=+1523.533938049" Oct 31 13:29:15.393309 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.389229 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:15.396745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.391016 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 13:29:15.396745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.396468 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 13:29:15.941682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.937893 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:15.941682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.938078 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 13:29:15.941682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:15.938124 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 13:29:16.309614 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:16.305059 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 13:29:16.309614 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:16.305134 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 13:29:16.344201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:16.344162 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 13:29:16.347861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:16.347825 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 13:29:16.417111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:16.417071 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 13:29:16.446665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:16.446622 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 13:29:16.918088 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:16.917756 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 31 13:29:16.935065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:16.934574 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 13:29:16.935065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:16.934647 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 13:29:17.224722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.206006 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:17.249121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.247914 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:17.331621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.323788 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" Oct 31 13:29:17.331621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.328740 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 13:29:17.331621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.328797 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 13:29:17.341258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.340708 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 13:29:17.341258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.340762 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 13:29:17.348618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.347637 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:17.415845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.387713 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:29:17.415845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.392976 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 13:29:17.415845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.393028 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 13:29:17.415845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.408235 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:29:17.415845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.409365 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:29:17.453733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.423696 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:17.453733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.425419 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" podStartSLOduration=4.425399418 podStartE2EDuration="4.425399418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:15.367959757 +0000 UTC m=+1523.513703653" watchObservedRunningTime="2026-10-31 13:29:17.425399418 +0000 UTC m=+1525.571143247" Oct 31 13:29:17.453733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.425587 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:17.540616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.526508 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-openshift-ca\") pod \"f93043d3-9e55-4d63-85e7-47e7e04076c0\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " Oct 31 13:29:17.581861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.581823 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-xks-ca\") pod \"f93043d3-9e55-4d63-85e7-47e7e04076c0\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " Oct 31 13:29:17.582614 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.582568 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93043d3-9e55-4d63-85e7-47e7e04076c0-kube-api-access-bq9q7\" (UniqueName: \"kubernetes.io/projected/f93043d3-9e55-4d63-85e7-47e7e04076c0-kube-api-access-bq9q7\") pod \"f93043d3-9e55-4d63-85e7-47e7e04076c0\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " Oct 31 13:29:17.591919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.538254 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-openshift-ca" pod "f93043d3-9e55-4d63-85e7-47e7e04076c0" (UID: "f93043d3-9e55-4d63-85e7-47e7e04076c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:17.801724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.700918 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:17.835152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.835109 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": context deadline exceeded" Oct 31 13:29:17.955929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.842153 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-xks-ca" pod "f93043d3-9e55-4d63-85e7-47e7e04076c0" (UID: "f93043d3-9e55-4d63-85e7-47e7e04076c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:29:17.956138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.858960 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93043d3-9e55-4d63-85e7-47e7e04076c0-kube-api-access-bq9q7" pod "f93043d3-9e55-4d63-85e7-47e7e04076c0" (UID: "f93043d3-9e55-4d63-85e7-47e7e04076c0"). InnerVolumeSpecName "kube-api-access-bq9q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:17.956257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.876411 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" event={"ID":"f93043d3-9e55-4d63-85e7-47e7e04076c0","Type":"ContainerDied","Data":"bc2aab2e822f18c9c42b4a7b153f9c978012a3ca7c3ba511c53f47139bbc3f98"} Oct 31 13:29:17.956407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.956387 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc2aab2e822f18c9c42b4a7b153f9c978012a3ca7c3ba511c53f47139bbc3f98" Oct 31 13:29:17.956496 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.876387 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8" Oct 31 13:29:17.967628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.963726 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:18.018628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.989470 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-xks-ca\") pod \"f93043d3-9e55-4d63-85e7-47e7e04076c0\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " Oct 31 13:29:18.018628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:17.989555 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93043d3-9e55-4d63-85e7-47e7e04076c0-kube-api-access-bq9q7\" (UniqueName: \"kubernetes.io/projected/f93043d3-9e55-4d63-85e7-47e7e04076c0-kube-api-access-bq9q7\") pod \"f93043d3-9e55-4d63-85e7-47e7e04076c0\" (UID: \"f93043d3-9e55-4d63-85e7-47e7e04076c0\") " Oct 31 13:29:18.022617 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:29:18.020215 5532 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/f93043d3-9e55-4d63-85e7-47e7e04076c0/volumes/kubernetes.io~secret/maas-api-xks-ca Oct 31 13:29:18.022617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.020244 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-xks-ca" pod "f93043d3-9e55-4d63-85e7-47e7e04076c0" (UID: "f93043d3-9e55-4d63-85e7-47e7e04076c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:29:18.022617 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:29:18.020310 5532 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/f93043d3-9e55-4d63-85e7-47e7e04076c0/volumes/kubernetes.io~projected/kube-api-access-bq9q7 Oct 31 13:29:18.022617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.020355 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93043d3-9e55-4d63-85e7-47e7e04076c0-kube-api-access-bq9q7" pod "f93043d3-9e55-4d63-85e7-47e7e04076c0" (UID: "f93043d3-9e55-4d63-85e7-47e7e04076c0"). InnerVolumeSpecName "kube-api-access-bq9q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:18.041678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.041639 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 13:29:18.045793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.045761 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 13:29:18.109678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.093229 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f93043d3-9e55-4d63-85e7-47e7e04076c0-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:18.208070 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.208033 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq9q7\" (UniqueName: \"kubernetes.io/projected/f93043d3-9e55-4d63-85e7-47e7e04076c0-kube-api-access-bq9q7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:18.208267 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.205670 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:29:18.209086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.209003 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:18.266527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.266482 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:29:18.301619 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:29:18.298352 5532 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93043d3_9e55_4d63_85e7_47e7e04076c0.slice\": RecentStats: unable to find data in memory cache]" Oct 31 13:29:18.407653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.324113 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 13:29:18.407951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.407918 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 13:29:18.435148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.387938 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:29:18.435464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.435435 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:29:18.436325 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.435637 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:29:18.436879 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.436586 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:29:18.437169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.437155 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:29:18.437363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.437338 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:29:18.443009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.350027 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:18.557825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.557775 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:29:18.633622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.349968 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 13:29:18.634279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.633870 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 13:29:18.634511 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.445845 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:18.634645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.448721 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:29:18.634823 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.634810 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:29:18.690670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.687423 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:29:18.690670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.435694 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:29:18.714108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.384567 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 13:29:18.714392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.714360 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 13:29:18.714517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.435660 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 13:29:18.727775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.727742 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:29:18.728011 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:18.727997 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:29:19.098676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.064753 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 13:29:19.098676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.064824 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 13:29:19.178875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.178661 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:19.251621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.232870 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:19.308629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.287705 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:29:19.825687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.339320 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:29:19.826363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.826341 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:29:19.889228 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.514036 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 13:29:19.889518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.889487 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 13:29:19.889660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.554178 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:19.954686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.520094 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:19.975557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.602776 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:29:20.088855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.800044 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:29:20.089194 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.800268 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:20.089428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.800468 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 13:29:20.089653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.504837 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:20.089850 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.554088 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 13:29:20.090039 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.089995 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 13:29:20.090195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.554250 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:29:20.091845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:19.726858 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 13:29:20.095702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.095660 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 13:29:20.170284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.163800 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:20.216950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.216862 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:20.547625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.493194 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:29:20.672627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.661204 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 13:29:20.677629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.672909 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 13:29:20.677855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.676782 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" podUID="3f952465-49ad-43cf-9598-dd3741e8de2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:20.830687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.830567 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:20.898871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.898834 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 13:29:20.899205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.899150 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 13:29:20.939645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.927796 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 13:29:20.939645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.927855 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 13:29:20.942008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.941962 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:29:21.002647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.997631 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 13:29:21.002647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.997686 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 13:29:21.002647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.997988 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 13:29:21.002647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:20.998020 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 13:29:21.002647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.001671 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:29:21.086696 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:29:21.084052 5532 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.048s" Oct 31 13:29:21.086696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.084133 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:29:21.092381 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.092341 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:21.108641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.098011 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:21.108641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.098046 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:21.108641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.108156 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:21.108641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.108422 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:29:21.109045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.109018 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:21.220941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.220887 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:21.416403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.415153 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 13:29:21.416403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.415211 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 13:29:21.425742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.425712 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 13:29:21.425949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.425923 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 13:29:21.426903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.426883 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:29:21.438574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.437186 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 13:29:21.438574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.437231 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 13:29:21.504662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.498149 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:21.908682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.893914 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:21.935701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.922793 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Oct 31 13:29:21.942945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.942905 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:21.946180 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.946152 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Oct 31 13:29:21.948380 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:21.948350 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Oct 31 13:29:22.006643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.004697 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:22.111480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.110714 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:22.111480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.111059 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:22.111480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.111266 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:22.129222 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.129183 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:22.134637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.131995 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:29:22.134637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.132060 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:29:22.134637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.132091 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:29:22.208501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.207332 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:29:22.292620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.259245 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:29:22.516909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.329364 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 13:29:22.516909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.329565 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Oct 31 13:29:22.516909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.516471 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Oct 31 13:29:22.516909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.516526 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 13:29:22.516909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.466741 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 13:29:22.516909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.516564 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 13:29:22.516909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.516603 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:29:22.516909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.516648 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:29:22.519367 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.519343 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:29:22.715474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.715443 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:29:22.719201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:22.719144 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:29:23.126760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:23.126321 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:23.128178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:23.128154 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:23.133662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:23.130350 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:29:23.326994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:23.326925 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Oct 31 13:29:23.326994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:23.327013 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Oct 31 13:29:23.374347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:23.374307 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 31 13:29:23.374525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:23.374372 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 31 13:29:23.787226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:23.787195 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:23.788825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:23.788779 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:23.795892 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:23.795827 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:29:24.331293 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:24.331261 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:24.331772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:24.331716 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:24.332431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:24.332412 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:29:24.397071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:24.397041 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:24.401618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:24.398724 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:24.405160 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:24.404381 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:29:29.010933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.010899 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j"] Oct 31 13:29:29.011651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.011631 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f93043d3-9e55-4d63-85e7-47e7e04076c0" containerName="revoke-keys" Oct 31 13:29:29.011651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.011649 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93043d3-9e55-4d63-85e7-47e7e04076c0" containerName="revoke-keys" Oct 31 13:29:29.011817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.011780 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93043d3-9e55-4d63-85e7-47e7e04076c0" containerName="revoke-keys" Oct 31 13:29:29.021126 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.021103 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.032433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.032409 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j"] Oct 31 13:29:29.127187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.127149 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.127362 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.127308 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.127362 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.127334 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.127534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.127516 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.127646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.127582 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.127738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.127715 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.127782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.127760 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwcnk\" (UniqueName: \"kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-kube-api-access-bwcnk\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.127819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.127785 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.127819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.127806 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.229156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.229123 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.229351 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.229166 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwcnk\" (UniqueName: \"kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-kube-api-access-bwcnk\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.229351 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.229193 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.229351 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.229210 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.229351 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.229247 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.229605 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.229360 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.229605 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.229385 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.229605 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.229546 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.229777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.229637 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.229777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.229697 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.229777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.229726 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-certs\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.229928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.229810 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-credential-socket\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.229928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.229896 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-data\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.230357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.230332 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istiod-ca-cert\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.231549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.231526 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-envoy\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.231854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.231836 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-podinfo\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.242069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.242041 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-token\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.242171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.242118 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwcnk\" (UniqueName: \"kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-kube-api-access-bwcnk\") pod \"e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.339792 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.339711 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:29.631058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.631025 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j"] Oct 31 13:29:29.631335 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:29:29.631306 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d1da6f9_c84f_4b0d_a805_7f3baa5b0cbd.slice/crio-e84fe3308f16344e0dc8d0ac096753cc4cc5412a56dfd0059c1a02bf0fed36d2 WatchSource:0}: Error finding container e84fe3308f16344e0dc8d0ac096753cc4cc5412a56dfd0059c1a02bf0fed36d2: Status 404 returned error can't find the container with id e84fe3308f16344e0dc8d0ac096753cc4cc5412a56dfd0059c1a02bf0fed36d2 Oct 31 13:29:29.634379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.634346 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:29.634486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.634420 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:29.634486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:29.634451 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:30.205274 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:30.205229 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" event={"ID":"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd","Type":"ContainerStarted","Data":"489352a06c6a12945cd6c093196654f693d8f171d20455c48047cc8ab7de1673"} Oct 31 13:29:30.206399 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:30.205337 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" event={"ID":"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd","Type":"ContainerStarted","Data":"e84fe3308f16344e0dc8d0ac096753cc4cc5412a56dfd0059c1a02bf0fed36d2"} Oct 31 13:29:30.233757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:30.233708 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podStartSLOduration=2.233693493 podStartE2EDuration="2.233693493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:30.230338306 +0000 UTC m=+1538.376082134" watchObservedRunningTime="2026-10-31 13:29:30.233693493 +0000 UTC m=+1538.379437320" Oct 31 13:29:30.355777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:30.355733 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:30.363937 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:30.363886 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:29:30.364268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:30.364229 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 13:29:31.363951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:31.363904 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:29:31.364364 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:31.363965 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 13:29:31.701322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:31.701287 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:29:31.788770 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:29:31.788727 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:29:31.874228 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:31.874184 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l"] Oct 31 13:29:31.878563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:31.878543 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" Oct 31 13:29:31.883769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:31.883749 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:29:31.894692 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:31.894659 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l"] Oct 31 13:29:31.984138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:31.984062 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:29:31.991007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:31.990979 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c41b41e-w5z6l\" (UID: \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" Oct 31 13:29:31.991120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:31.991092 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c41b41e-w5z6l\" (UID: \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" Oct 31 13:29:31.991167 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:31.991127 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbc4b\" (UniqueName: \"kubernetes.io/projected/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-kube-api-access-hbc4b\") pod \"maas-api-revoke-sub-7c41b41e-w5z6l\" (UID: \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" Oct 31 13:29:32.093142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:32.093103 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c41b41e-w5z6l\" (UID: \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" Oct 31 13:29:32.093394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:32.093375 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbc4b\" (UniqueName: \"kubernetes.io/projected/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-kube-api-access-hbc4b\") pod \"maas-api-revoke-sub-7c41b41e-w5z6l\" (UID: \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" Oct 31 13:29:32.093865 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:32.093845 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c41b41e-w5z6l\" (UID: \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" Oct 31 13:29:32.094719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:32.094681 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c41b41e-w5z6l\" (UID: \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" Oct 31 13:29:32.096772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:32.096747 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c41b41e-w5z6l\" (UID: \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" Oct 31 13:29:32.107380 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:32.107349 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbc4b\" (UniqueName: \"kubernetes.io/projected/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-kube-api-access-hbc4b\") pod \"maas-api-revoke-sub-7c41b41e-w5z6l\" (UID: \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" Oct 31 13:29:32.195761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:32.195725 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" Oct 31 13:29:32.353570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:32.351981 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:29:32.353570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:32.352038 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 13:29:32.582702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:32.582364 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l"] Oct 31 13:29:33.355370 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:33.355283 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:29:33.359652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:33.357579 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" event={"ID":"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce","Type":"ContainerStarted","Data":"c370d6987978092af6f5c18f9c7af99c9a3160add1de3a70177e46801461732d"} Oct 31 13:29:33.374126 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:33.374088 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" event={"ID":"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce","Type":"ContainerStarted","Data":"0247ad014fd3a23247385d90b53758543566d84b7b31c184b8afa5bcf0c3dcd6"} Oct 31 13:29:33.374365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:33.358743 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 13:29:33.470897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:33.470835 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" podStartSLOduration=2.470814909 podStartE2EDuration="2.470814909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:33.433950258 +0000 UTC m=+1541.579694086" watchObservedRunningTime="2026-10-31 13:29:33.470814909 +0000 UTC m=+1541.616558738" Oct 31 13:29:34.351866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:34.351829 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:29:34.352460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:34.352427 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 13:29:34.422005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:34.421395 5532 generic.go:356] "Generic (PLEG): container finished" podID="e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce" containerID="c370d6987978092af6f5c18f9c7af99c9a3160add1de3a70177e46801461732d" exitCode=0 Oct 31 13:29:34.422005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:34.421609 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" event={"ID":"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce","Type":"ContainerDied","Data":"c370d6987978092af6f5c18f9c7af99c9a3160add1de3a70177e46801461732d"} Oct 31 13:29:34.655888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:34.654761 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": context deadline exceeded" Oct 31 13:29:34.902620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:34.897098 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:34.902620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:34.900940 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:35.123675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.103340 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:35.123675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.110878 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:35.123675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.111633 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:35.123675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.111921 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:29:35.156623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.125023 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:35.156623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.125119 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Oct 31 13:29:35.156623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.125168 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:29:35.156623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.134320 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:35.255620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.249691 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:29:35.423678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.422238 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:29:35.423678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.422295 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 13:29:35.466322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.424500 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded" Oct 31 13:29:35.511936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.485688 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:29:35.511936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.485771 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:35.532939 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.532508 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:35.948964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.948916 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:29:35.970149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.969698 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:29:35.970752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.949193 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:29:35.971614 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:35.970951 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:29:36.127973 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.127929 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:29:36.128444 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.128413 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:29:36.130928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.130902 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:29:36.137692 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.137662 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:36.148619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.142654 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:36.166006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.165964 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:36.167627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.167577 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:29:36.251812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.251706 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:36.294632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.292778 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:29:36.294632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.292852 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:29:36.294632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.292894 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:29:36.294632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.292930 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:29:36.294632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.292972 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:29:36.294632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.293010 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:29:36.294632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.293047 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:29:36.294632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.293083 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:29:36.389812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.389762 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:29:36.390141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.390091 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 13:29:36.584534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.561854 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:36.599520 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:36.599486 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:29:37.020657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.015468 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:29:37.172408 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.172367 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:37.173269 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.173238 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:37.198895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.198843 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:37.199851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.199815 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:37.426142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.202771 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:37.426142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.208487 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:29:37.426142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.208711 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:29:37.426142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.208909 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:29:37.426142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.216119 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 31 13:29:37.426142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.260376 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:29:37.601491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.596792 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:29:37.663774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.663717 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:37.830677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.769708 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:37.830677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.769828 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:37.946621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:37.868671 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 13:29:38.173968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.060174 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:29:38.602631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.197831 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:38.602631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.220785 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:29:38.602631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.593124 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" podUID="7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:38.602631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.297651 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 13:29:38.602631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.221912 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:38.626619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.455046 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:29:38.626619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.612198 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:38.626619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.483461 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:29:38.626619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.483343 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:29:38.626619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.483429 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:29:38.626619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.483513 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:29:38.626619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.550971 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:29:38.626619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.621136 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 13:29:38.626619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.577573 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:29:38.626619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.621205 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" podUID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:38.626619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.483483 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:29:38.645618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.552329 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:38.683000 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.682908 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:29:38.683258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.683227 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" podUID="0a27b459-9ad5-4ade-87f3-6ef3ef735e30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:38.684573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.483404 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:29:38.734619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.709268 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" Oct 31 13:29:38.766010 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.717693 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:29:38.766304 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.766271 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" podUID="5a9332ef-beec-4b46-9617-36d8ce8812c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:38.766500 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.726842 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:38.958674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.944125 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-openshift-ca\") pod \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\" (UID: \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\") " Oct 31 13:29:38.958674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.944351 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-kube-api-access-hbc4b\" (UniqueName: \"kubernetes.io/projected/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-kube-api-access-hbc4b\") pod \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\" (UID: \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\") " Oct 31 13:29:38.958674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.944441 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-xks-ca\") pod \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\" (UID: \"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce\") " Oct 31 13:29:38.974888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:38.974848 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-openshift-ca" pod "e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce" (UID: "e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:39.060637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.034073 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" event={"ID":"e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce","Type":"ContainerDied","Data":"0247ad014fd3a23247385d90b53758543566d84b7b31c184b8afa5bcf0c3dcd6"} Oct 31 13:29:39.060637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.034152 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0247ad014fd3a23247385d90b53758543566d84b7b31c184b8afa5bcf0c3dcd6" Oct 31 13:29:39.060637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.034243 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l" Oct 31 13:29:39.127664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.111747 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-xks-ca" pod "e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce" (UID: "e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:29:39.127664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.115576 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:39.127664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.115633 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:39.225668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.224653 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-kube-api-access-hbc4b" pod "e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce" (UID: "e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce"). InnerVolumeSpecName "kube-api-access-hbc4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:39.233619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.226329 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbc4b\" (UniqueName: \"kubernetes.io/projected/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce-kube-api-access-hbc4b\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:39.296618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.293770 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:29:39.296618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.293835 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:39.364738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.364700 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:29:39.365005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.364977 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 13:29:39.368674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.365947 5532 patch_prober.go:29] interesting pod/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:29:39.372615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.368838 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" podUID="48476992-f75d-4caf-aed3-02cd8c339644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:39.585674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.575579 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:29:39.591763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.591722 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:29:39.603805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.601252 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:29:39.603805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.601692 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 13:29:39.649773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.649730 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:29:39.650007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.649979 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:39.650157 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.650135 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:29:39.650616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.650288 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:39.650732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.650349 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:29:39.699205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.699163 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:29:39.768675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.767918 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:39.820615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.803744 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:29:39.837968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:39.837870 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:40.348580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.347645 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:29:40.348580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.347711 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 13:29:40.593269 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.589697 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:40.593269 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.592507 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:40.619108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.619008 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:40.619858 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.619707 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:29:40.658264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.658223 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:29:40.659009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.658510 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:40.659915 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.658539 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:29:40.669430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.658574 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:29:40.692455 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.658614 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:29:40.769033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.768128 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:29:40.900102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.877196 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:29:40.995361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:40.995329 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:29:41.011820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:41.011794 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:29:41.012615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:41.012571 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:29:41.036736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:41.035396 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:29:41.048228 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:41.048193 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:29:41.065518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:41.065493 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:29:41.065831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:41.065812 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:29:41.069022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:41.069003 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:29:41.073334 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:41.073314 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:29:41.340387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:41.340310 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Oct 31 13:29:41.340387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:41.340361 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Oct 31 13:29:43.380770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:43.380732 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:29:43.381369 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:43.381333 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:29:43.512022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:43.511993 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:43.512485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:43.512443 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:43.513466 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:43.513443 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:29:45.553612 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.553562 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf"] Oct 31 13:29:45.554271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.554254 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce" containerName="revoke-keys" Oct 31 13:29:45.554318 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.554271 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce" containerName="revoke-keys" Oct 31 13:29:45.554411 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.554401 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce" containerName="revoke-keys" Oct 31 13:29:45.559185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.559168 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" Oct 31 13:29:45.564009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.563979 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:29:45.564214 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.564193 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:29:45.567345 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.567318 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf"] Oct 31 13:29:45.631548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.631512 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stf4q\" (UniqueName: \"kubernetes.io/projected/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-kube-api-access-stf4q\") pod \"maas-api-revoke-sub-883db104-qhbxf\" (UID: \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" Oct 31 13:29:45.631821 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.631798 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-883db104-qhbxf\" (UID: \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" Oct 31 13:29:45.632177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.632156 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-883db104-qhbxf\" (UID: \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" Oct 31 13:29:45.734148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.734115 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stf4q\" (UniqueName: \"kubernetes.io/projected/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-kube-api-access-stf4q\") pod \"maas-api-revoke-sub-883db104-qhbxf\" (UID: \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" Oct 31 13:29:45.734352 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.734193 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-883db104-qhbxf\" (UID: \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" Oct 31 13:29:45.734505 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.734486 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-883db104-qhbxf\" (UID: \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" Oct 31 13:29:45.735095 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.735068 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-883db104-qhbxf\" (UID: \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" Oct 31 13:29:45.737185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.737158 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-883db104-qhbxf\" (UID: \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" Oct 31 13:29:45.750497 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.750471 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stf4q\" (UniqueName: \"kubernetes.io/projected/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-kube-api-access-stf4q\") pod \"maas-api-revoke-sub-883db104-qhbxf\" (UID: \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" Oct 31 13:29:45.875760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:45.875420 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" Oct 31 13:29:46.160421 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:29:46.160395 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2b78bfc_ece6_422f_838b_f6c8c7f2425a.slice/crio-29e03dc8f8b1f08933b67d87562611cf0464f99849597a7c2b9b83cab20970eb WatchSource:0}: Error finding container 29e03dc8f8b1f08933b67d87562611cf0464f99849597a7c2b9b83cab20970eb: Status 404 returned error can't find the container with id 29e03dc8f8b1f08933b67d87562611cf0464f99849597a7c2b9b83cab20970eb Oct 31 13:29:46.160645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:46.160620 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf"] Oct 31 13:29:46.214763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:46.214719 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" event={"ID":"c2b78bfc-ece6-422f-838b-f6c8c7f2425a","Type":"ContainerStarted","Data":"29e03dc8f8b1f08933b67d87562611cf0464f99849597a7c2b9b83cab20970eb"} Oct 31 13:29:47.228022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:47.227986 5532 generic.go:356] "Generic (PLEG): container finished" podID="c2b78bfc-ece6-422f-838b-f6c8c7f2425a" containerID="d0000a8cf098e0d9596a57e8c74f96722c6bdcbebaa4b427dbeb13cd9ca97bfd" exitCode=0 Oct 31 13:29:47.228431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:47.228074 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" event={"ID":"c2b78bfc-ece6-422f-838b-f6c8c7f2425a","Type":"ContainerDied","Data":"d0000a8cf098e0d9596a57e8c74f96722c6bdcbebaa4b427dbeb13cd9ca97bfd"} Oct 31 13:29:47.952868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:47.952831 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9"] Oct 31 13:29:47.957651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:47.957633 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" Oct 31 13:29:47.967694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:47.967667 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9"] Oct 31 13:29:48.068353 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.068311 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0e79e99-thmt9\" (UID: \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" Oct 31 13:29:48.068353 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.068360 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scwbk\" (UniqueName: \"kubernetes.io/projected/e5afab8a-9529-4205-98b3-8ea642c7ddeb-kube-api-access-scwbk\") pod \"maas-api-revoke-sub-d0e79e99-thmt9\" (UID: \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" Oct 31 13:29:48.068620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.068576 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0e79e99-thmt9\" (UID: \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" Oct 31 13:29:48.170808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.170768 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0e79e99-thmt9\" (UID: \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" Oct 31 13:29:48.170998 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.170840 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scwbk\" (UniqueName: \"kubernetes.io/projected/e5afab8a-9529-4205-98b3-8ea642c7ddeb-kube-api-access-scwbk\") pod \"maas-api-revoke-sub-d0e79e99-thmt9\" (UID: \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" Oct 31 13:29:48.171119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.171072 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0e79e99-thmt9\" (UID: \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" Oct 31 13:29:48.171584 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.171556 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0e79e99-thmt9\" (UID: \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" Oct 31 13:29:48.173411 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.173390 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0e79e99-thmt9\" (UID: \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" Oct 31 13:29:48.184589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.184564 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scwbk\" (UniqueName: \"kubernetes.io/projected/e5afab8a-9529-4205-98b3-8ea642c7ddeb-kube-api-access-scwbk\") pod \"maas-api-revoke-sub-d0e79e99-thmt9\" (UID: \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" Oct 31 13:29:48.275065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.275027 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" Oct 31 13:29:48.438546 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.438187 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" Oct 31 13:29:48.577846 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.577736 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-kube-api-access-stf4q\" (UniqueName: \"kubernetes.io/projected/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-kube-api-access-stf4q\") pod \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\" (UID: \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\") " Oct 31 13:29:48.578064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.577938 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-openshift-ca\") pod \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\" (UID: \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\") " Oct 31 13:29:48.578064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.577979 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-xks-ca\") pod \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\" (UID: \"c2b78bfc-ece6-422f-838b-f6c8c7f2425a\") " Oct 31 13:29:48.578359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.578331 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-openshift-ca" pod "c2b78bfc-ece6-422f-838b-f6c8c7f2425a" (UID: "c2b78bfc-ece6-422f-838b-f6c8c7f2425a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:48.578621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.578578 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:48.580453 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.580431 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-xks-ca" pod "c2b78bfc-ece6-422f-838b-f6c8c7f2425a" (UID: "c2b78bfc-ece6-422f-838b-f6c8c7f2425a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:29:48.580526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.580484 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-kube-api-access-stf4q" pod "c2b78bfc-ece6-422f-838b-f6c8c7f2425a" (UID: "c2b78bfc-ece6-422f-838b-f6c8c7f2425a"). InnerVolumeSpecName "kube-api-access-stf4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:48.601870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.601841 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9"] Oct 31 13:29:48.603457 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:29:48.603429 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5afab8a_9529_4205_98b3_8ea642c7ddeb.slice/crio-2c62e5634a89568ce840981d278d400d1afb3ea8403f932b64bfdb53460507c4 WatchSource:0}: Error finding container 2c62e5634a89568ce840981d278d400d1afb3ea8403f932b64bfdb53460507c4: Status 404 returned error can't find the container with id 2c62e5634a89568ce840981d278d400d1afb3ea8403f932b64bfdb53460507c4 Oct 31 13:29:48.680446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.680412 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:48.680446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:48.680443 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stf4q\" (UniqueName: \"kubernetes.io/projected/c2b78bfc-ece6-422f-838b-f6c8c7f2425a-kube-api-access-stf4q\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:49.245401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:49.245348 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" event={"ID":"e5afab8a-9529-4205-98b3-8ea642c7ddeb","Type":"ContainerStarted","Data":"f340ad496f66f9008e523a38bbb6f658690dd3bbd868952ac087c86acf8a00d7"} Oct 31 13:29:49.245587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:49.245431 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" event={"ID":"e5afab8a-9529-4205-98b3-8ea642c7ddeb","Type":"ContainerStarted","Data":"2c62e5634a89568ce840981d278d400d1afb3ea8403f932b64bfdb53460507c4"} Oct 31 13:29:49.246947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:49.246920 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" Oct 31 13:29:49.246947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:49.246936 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf" event={"ID":"c2b78bfc-ece6-422f-838b-f6c8c7f2425a","Type":"ContainerDied","Data":"29e03dc8f8b1f08933b67d87562611cf0464f99849597a7c2b9b83cab20970eb"} Oct 31 13:29:49.247104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:49.246976 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29e03dc8f8b1f08933b67d87562611cf0464f99849597a7c2b9b83cab20970eb" Oct 31 13:29:49.271709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:49.271657 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" podStartSLOduration=2.271640304 podStartE2EDuration="2.271640304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:49.267433053 +0000 UTC m=+1557.413176881" watchObservedRunningTime="2026-10-31 13:29:49.271640304 +0000 UTC m=+1557.417384133" Oct 31 13:29:50.255542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:50.255510 5532 generic.go:356] "Generic (PLEG): container finished" podID="e5afab8a-9529-4205-98b3-8ea642c7ddeb" containerID="f340ad496f66f9008e523a38bbb6f658690dd3bbd868952ac087c86acf8a00d7" exitCode=0 Oct 31 13:29:50.255965 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:50.255631 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" event={"ID":"e5afab8a-9529-4205-98b3-8ea642c7ddeb","Type":"ContainerDied","Data":"f340ad496f66f9008e523a38bbb6f658690dd3bbd868952ac087c86acf8a00d7"} Oct 31 13:29:51.455170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:51.455141 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" Oct 31 13:29:51.527898 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:51.527813 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5afab8a-9529-4205-98b3-8ea642c7ddeb-kube-api-access-scwbk\" (UniqueName: \"kubernetes.io/projected/e5afab8a-9529-4205-98b3-8ea642c7ddeb-kube-api-access-scwbk\") pod \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\" (UID: \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\") " Oct 31 13:29:51.528044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:51.527985 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-openshift-ca\") pod \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\" (UID: \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\") " Oct 31 13:29:51.528131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:51.528115 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-xks-ca\") pod \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\" (UID: \"e5afab8a-9529-4205-98b3-8ea642c7ddeb\") " Oct 31 13:29:51.528392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:51.528355 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-openshift-ca" pod "e5afab8a-9529-4205-98b3-8ea642c7ddeb" (UID: "e5afab8a-9529-4205-98b3-8ea642c7ddeb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:51.528802 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:51.528778 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:51.530039 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:51.530001 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5afab8a-9529-4205-98b3-8ea642c7ddeb-kube-api-access-scwbk" pod "e5afab8a-9529-4205-98b3-8ea642c7ddeb" (UID: "e5afab8a-9529-4205-98b3-8ea642c7ddeb"). InnerVolumeSpecName "kube-api-access-scwbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:51.530171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:51.530126 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-xks-ca" pod "e5afab8a-9529-4205-98b3-8ea642c7ddeb" (UID: "e5afab8a-9529-4205-98b3-8ea642c7ddeb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:29:51.630251 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:51.630206 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5afab8a-9529-4205-98b3-8ea642c7ddeb-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:51.630461 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:51.630286 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scwbk\" (UniqueName: \"kubernetes.io/projected/e5afab8a-9529-4205-98b3-8ea642c7ddeb-kube-api-access-scwbk\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:52.275560 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:52.275484 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" Oct 31 13:29:52.275703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:52.275486 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9" event={"ID":"e5afab8a-9529-4205-98b3-8ea642c7ddeb","Type":"ContainerDied","Data":"2c62e5634a89568ce840981d278d400d1afb3ea8403f932b64bfdb53460507c4"} Oct 31 13:29:52.275703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:52.275636 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c62e5634a89568ce840981d278d400d1afb3ea8403f932b64bfdb53460507c4" Oct 31 13:29:56.090562 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.090516 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm"] Oct 31 13:29:56.091672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.091633 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c2b78bfc-ece6-422f-838b-f6c8c7f2425a" containerName="revoke-keys" Oct 31 13:29:56.091672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.091661 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2b78bfc-ece6-422f-838b-f6c8c7f2425a" containerName="revoke-keys" Oct 31 13:29:56.091869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.091734 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e5afab8a-9529-4205-98b3-8ea642c7ddeb" containerName="revoke-keys" Oct 31 13:29:56.091869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.091744 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5afab8a-9529-4205-98b3-8ea642c7ddeb" containerName="revoke-keys" Oct 31 13:29:56.091976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.091939 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5afab8a-9529-4205-98b3-8ea642c7ddeb" containerName="revoke-keys" Oct 31 13:29:56.091976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.091959 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2b78bfc-ece6-422f-838b-f6c8c7f2425a" containerName="revoke-keys" Oct 31 13:29:56.110066 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.099033 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" Oct 31 13:29:56.110066 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.107350 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm"] Oct 31 13:29:56.110066 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.107576 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:29:56.123699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.111811 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:29:56.206822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.206785 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68zks\" (UniqueName: \"kubernetes.io/projected/301589df-3ff3-4a56-a4cf-7ca0e1471852-kube-api-access-68zks\") pod \"maas-api-revoke-sub-0b06227e-z5gqm\" (UID: \"301589df-3ff3-4a56-a4cf-7ca0e1471852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" Oct 31 13:29:56.206992 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.206854 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b06227e-z5gqm\" (UID: \"301589df-3ff3-4a56-a4cf-7ca0e1471852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" Oct 31 13:29:56.207104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.207080 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b06227e-z5gqm\" (UID: \"301589df-3ff3-4a56-a4cf-7ca0e1471852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" Oct 31 13:29:56.308921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.308890 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68zks\" (UniqueName: \"kubernetes.io/projected/301589df-3ff3-4a56-a4cf-7ca0e1471852-kube-api-access-68zks\") pod \"maas-api-revoke-sub-0b06227e-z5gqm\" (UID: \"301589df-3ff3-4a56-a4cf-7ca0e1471852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" Oct 31 13:29:56.309139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.308951 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b06227e-z5gqm\" (UID: \"301589df-3ff3-4a56-a4cf-7ca0e1471852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" Oct 31 13:29:56.309139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.309049 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b06227e-z5gqm\" (UID: \"301589df-3ff3-4a56-a4cf-7ca0e1471852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" Oct 31 13:29:56.309662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.309643 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b06227e-z5gqm\" (UID: \"301589df-3ff3-4a56-a4cf-7ca0e1471852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" Oct 31 13:29:56.311465 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.311440 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b06227e-z5gqm\" (UID: \"301589df-3ff3-4a56-a4cf-7ca0e1471852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" Oct 31 13:29:56.322105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.322075 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68zks\" (UniqueName: \"kubernetes.io/projected/301589df-3ff3-4a56-a4cf-7ca0e1471852-kube-api-access-68zks\") pod \"maas-api-revoke-sub-0b06227e-z5gqm\" (UID: \"301589df-3ff3-4a56-a4cf-7ca0e1471852\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" Oct 31 13:29:56.452948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.452917 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" Oct 31 13:29:56.744499 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:29:56.744462 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod301589df_3ff3_4a56_a4cf_7ca0e1471852.slice/crio-b89f36e08d1632fefea379925b6e3a6997d96f0ed87fe7561e1c800def20d4ed WatchSource:0}: Error finding container b89f36e08d1632fefea379925b6e3a6997d96f0ed87fe7561e1c800def20d4ed: Status 404 returned error can't find the container with id b89f36e08d1632fefea379925b6e3a6997d96f0ed87fe7561e1c800def20d4ed Oct 31 13:29:56.745059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:56.745035 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm"] Oct 31 13:29:57.320062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:57.319963 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" event={"ID":"301589df-3ff3-4a56-a4cf-7ca0e1471852","Type":"ContainerStarted","Data":"d3530bca24c5a18430ab471fa1505520d885e2a04afedaa1b45ac570e60485fd"} Oct 31 13:29:57.320062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:57.320040 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" event={"ID":"301589df-3ff3-4a56-a4cf-7ca0e1471852","Type":"ContainerStarted","Data":"b89f36e08d1632fefea379925b6e3a6997d96f0ed87fe7561e1c800def20d4ed"} Oct 31 13:29:57.354490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:57.354434 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" podStartSLOduration=1.354419306 podStartE2EDuration="1.354419306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:29:57.35267176 +0000 UTC m=+1565.498415592" watchObservedRunningTime="2026-10-31 13:29:57.354419306 +0000 UTC m=+1565.500163133" Oct 31 13:29:58.329664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.329630 5532 generic.go:356] "Generic (PLEG): container finished" podID="301589df-3ff3-4a56-a4cf-7ca0e1471852" containerID="d3530bca24c5a18430ab471fa1505520d885e2a04afedaa1b45ac570e60485fd" exitCode=0 Oct 31 13:29:58.330046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.329724 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" event={"ID":"301589df-3ff3-4a56-a4cf-7ca0e1471852","Type":"ContainerDied","Data":"d3530bca24c5a18430ab471fa1505520d885e2a04afedaa1b45ac570e60485fd"} Oct 31 13:29:58.763776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.763740 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5"] Oct 31 13:29:58.768722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.768695 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.825101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.821065 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5"] Oct 31 13:29:58.833904 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.832503 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb"] Oct 31 13:29:58.839569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.839543 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:58.864358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.864066 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.864358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.864146 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.864358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.864278 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.864358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.864354 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmh2r\" (UniqueName: \"kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-kube-api-access-lmh2r\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.864642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.864455 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.864642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.864507 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.864642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.864615 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.864748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.864654 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.864794 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.864775 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.889608 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.889550 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb"] Oct 31 13:29:58.966501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.966463 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd574\" (UniqueName: \"kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-kube-api-access-fd574\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:58.966699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.966518 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761abd89-b630-42f8-91bc-6096ef23d6a4-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:58.966699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.966609 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.966786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.966719 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:58.966786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.966750 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:58.966853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.966810 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.966853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.966841 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.966915 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.966884 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.966951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.966915 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:58.966951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.966933 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:58.967062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.967040 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.967102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.967064 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:58.967149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.967135 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmh2r\" (UniqueName: \"kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-kube-api-access-lmh2r\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.967212 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.967165 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:58.967248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.967218 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.967292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.967243 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.967292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.967257 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.967375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.967313 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.967425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.967415 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.967462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.967443 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.967513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.967482 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:58.967755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.967735 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.968109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.968084 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.969585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.969561 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.969687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.969649 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.980007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.979976 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:58.980124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:58.980056 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmh2r\" (UniqueName: \"kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-kube-api-access-lmh2r\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:59.069292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.069198 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.069292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.069268 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd574\" (UniqueName: \"kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-kube-api-access-fd574\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.069521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.069369 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761abd89-b630-42f8-91bc-6096ef23d6a4-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.069521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.069477 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.069521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.069496 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.069724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.069566 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.069724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.069590 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.069724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.069650 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.069724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.069692 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.069724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.069714 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.069977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.069850 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-certs\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.070035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.070022 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-credential-socket\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.070090 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.070024 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761abd89-b630-42f8-91bc-6096ef23d6a4-istiod-ca-cert\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.070140 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.070092 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-data\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.071891 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.071866 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-envoy\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.072211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.072189 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-podinfo\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.082292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.082266 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd574\" (UniqueName: \"kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-kube-api-access-fd574\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.082483 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.082462 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-token\") pod \"e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.121835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.121800 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:29:59.172688 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.172651 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5"] Oct 31 13:29:59.177555 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.177524 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:29:59.179490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.179465 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.208206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.208170 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5"] Oct 31 13:29:59.273549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.273500 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.273754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.273660 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.273754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.273729 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgvqw\" (UniqueName: \"kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-kube-api-access-cgvqw\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.273891 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.273769 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.273891 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.273818 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.273891 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.273847 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.274059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.273976 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.274059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.274010 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf67cc4-ed1a-4169-957b-0fe690432386-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.274059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.274048 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.376088 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.376052 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.377761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.376220 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.377761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.376287 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.377761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.376357 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgvqw\" (UniqueName: \"kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-kube-api-access-cgvqw\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.377761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.376388 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.377761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.376422 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.377761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.376459 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.377761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.376504 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-data\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.377761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.377076 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.377761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.377382 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-credential-socket\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.377761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.377527 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.377761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.377578 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-certs\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.377761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.377585 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf67cc4-ed1a-4169-957b-0fe690432386-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.378454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.378369 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf67cc4-ed1a-4169-957b-0fe690432386-istiod-ca-cert\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.381447 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.381395 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-podinfo\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.381840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.381805 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-envoy\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.391280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.391253 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgvqw\" (UniqueName: \"kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-kube-api-access-cgvqw\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.392847 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.392800 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-token\") pod \"e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.432619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.432498 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5"] Oct 31 13:29:59.442167 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:29:59.442138 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb8d480_1daf_41eb_aab7_e988dc35c2d6.slice/crio-76a5752d72c5809913a266b2a25c5cbecc7424ca96e581462015d3383d18f406 WatchSource:0}: Error finding container 76a5752d72c5809913a266b2a25c5cbecc7424ca96e581462015d3383d18f406: Status 404 returned error can't find the container with id 76a5752d72c5809913a266b2a25c5cbecc7424ca96e581462015d3383d18f406 Oct 31 13:29:59.446335 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.446270 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:59.446451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.446425 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:59.446534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.446514 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:59.497487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.497462 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" Oct 31 13:29:59.541943 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.540166 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:29:59.543120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.543089 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb"] Oct 31 13:29:59.543922 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:29:59.543898 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761abd89_b630_42f8_91bc_6096ef23d6a4.slice/crio-ce6f5ea5165bc2e7a49d95e234b77738dff2587372c0b72c0bbe5c43686fa108 WatchSource:0}: Error finding container ce6f5ea5165bc2e7a49d95e234b77738dff2587372c0b72c0bbe5c43686fa108: Status 404 returned error can't find the container with id ce6f5ea5165bc2e7a49d95e234b77738dff2587372c0b72c0bbe5c43686fa108 Oct 31 13:29:59.546725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.546691 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:59.546828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.546766 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:59.546828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.546821 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:59.581391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.581346 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-openshift-ca\") pod \"301589df-3ff3-4a56-a4cf-7ca0e1471852\" (UID: \"301589df-3ff3-4a56-a4cf-7ca0e1471852\") " Oct 31 13:29:59.581895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.581846 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/301589df-3ff3-4a56-a4cf-7ca0e1471852-kube-api-access-68zks\" (UniqueName: \"kubernetes.io/projected/301589df-3ff3-4a56-a4cf-7ca0e1471852-kube-api-access-68zks\") pod \"301589df-3ff3-4a56-a4cf-7ca0e1471852\" (UID: \"301589df-3ff3-4a56-a4cf-7ca0e1471852\") " Oct 31 13:29:59.582050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.582008 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-xks-ca\") pod \"301589df-3ff3-4a56-a4cf-7ca0e1471852\" (UID: \"301589df-3ff3-4a56-a4cf-7ca0e1471852\") " Oct 31 13:29:59.586833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.585769 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-xks-ca" pod "301589df-3ff3-4a56-a4cf-7ca0e1471852" (UID: "301589df-3ff3-4a56-a4cf-7ca0e1471852"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:29:59.589299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.589273 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-openshift-ca" pod "301589df-3ff3-4a56-a4cf-7ca0e1471852" (UID: "301589df-3ff3-4a56-a4cf-7ca0e1471852"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:29:59.589426 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.589306 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/301589df-3ff3-4a56-a4cf-7ca0e1471852-kube-api-access-68zks" pod "301589df-3ff3-4a56-a4cf-7ca0e1471852" (UID: "301589df-3ff3-4a56-a4cf-7ca0e1471852"). InnerVolumeSpecName "kube-api-access-68zks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:29:59.685676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.685551 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68zks\" (UniqueName: \"kubernetes.io/projected/301589df-3ff3-4a56-a4cf-7ca0e1471852-kube-api-access-68zks\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:59.685929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.685586 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:59.685929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.685907 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/301589df-3ff3-4a56-a4cf-7ca0e1471852-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:29:59.874254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.874219 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5"] Oct 31 13:29:59.876360 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:29:59.876331 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bf67cc4_ed1a_4169_957b_0fe690432386.slice/crio-8e1cee708f2adcf3ed027ea8c47fe029ef59b15e05a1eadb78ecd9e3ee737b03 WatchSource:0}: Error finding container 8e1cee708f2adcf3ed027ea8c47fe029ef59b15e05a1eadb78ecd9e3ee737b03: Status 404 returned error can't find the container with id 8e1cee708f2adcf3ed027ea8c47fe029ef59b15e05a1eadb78ecd9e3ee737b03 Oct 31 13:29:59.879922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.879878 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:59.880009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.879972 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:29:59.880056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:29:59.880024 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:30:00.171493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.168710 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26"] Oct 31 13:30:00.171493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.169816 5532 cpu_manager.go:414] "containerMap: removing container" podUID="301589df-3ff3-4a56-a4cf-7ca0e1471852" containerName="revoke-keys" Oct 31 13:30:00.171493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.169839 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="301589df-3ff3-4a56-a4cf-7ca0e1471852" containerName="revoke-keys" Oct 31 13:30:00.171493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.170004 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="301589df-3ff3-4a56-a4cf-7ca0e1471852" containerName="revoke-keys" Oct 31 13:30:00.185750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.184222 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" Oct 31 13:30:00.200614 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.197326 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q"] Oct 31 13:30:00.206370 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.206335 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" Oct 31 13:30:00.210645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.210179 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2"] Oct 31 13:30:00.225496 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.225307 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" Oct 31 13:30:00.269031 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.268936 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26"] Oct 31 13:30:00.279143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.279059 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q"] Oct 31 13:30:00.286919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.286682 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw"] Oct 31 13:30:00.311702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.311084 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f66256-f121-47a7-b1a9-f6cb089802c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q\" (UID: \"08f66256-f121-47a7-b1a9-f6cb089802c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" Oct 31 13:30:00.311702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.311140 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l55ff\" (UniqueName: \"kubernetes.io/projected/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-kube-api-access-l55ff\") pod \"maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26\" (UID: \"2bcf7019-e8dc-4f70-a37c-3e2640022b8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" Oct 31 13:30:00.311702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.311290 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlqw5\" (UniqueName: \"kubernetes.io/projected/08f66256-f121-47a7-b1a9-f6cb089802c0-kube-api-access-jlqw5\") pod \"maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q\" (UID: \"08f66256-f121-47a7-b1a9-f6cb089802c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" Oct 31 13:30:00.311702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.311516 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26\" (UID: \"2bcf7019-e8dc-4f70-a37c-3e2640022b8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" Oct 31 13:30:00.313221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.313196 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" Oct 31 13:30:00.406492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.406450 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" event={"ID":"8bf67cc4-ed1a-4169-957b-0fe690432386","Type":"ContainerStarted","Data":"7a1a6c45a407e2a5cd51815da7caf99dc68d8aaf53fcba4622627e0ea5836467"} Oct 31 13:30:00.407926 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.407883 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" event={"ID":"8bf67cc4-ed1a-4169-957b-0fe690432386","Type":"ContainerStarted","Data":"8e1cee708f2adcf3ed027ea8c47fe029ef59b15e05a1eadb78ecd9e3ee737b03"} Oct 31 13:30:00.417143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.416426 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f66256-f121-47a7-b1a9-f6cb089802c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q\" (UID: \"08f66256-f121-47a7-b1a9-f6cb089802c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" Oct 31 13:30:00.417143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.416476 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l55ff\" (UniqueName: \"kubernetes.io/projected/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-kube-api-access-l55ff\") pod \"maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26\" (UID: \"2bcf7019-e8dc-4f70-a37c-3e2640022b8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" Oct 31 13:30:00.417143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.416627 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlqw5\" (UniqueName: \"kubernetes.io/projected/08f66256-f121-47a7-b1a9-f6cb089802c0-kube-api-access-jlqw5\") pod \"maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q\" (UID: \"08f66256-f121-47a7-b1a9-f6cb089802c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" Oct 31 13:30:00.417143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.416784 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw\" (UID: \"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" Oct 31 13:30:00.417143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.416833 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78p5c\" (UniqueName: \"kubernetes.io/projected/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-kube-api-access-78p5c\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw\" (UID: \"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" Oct 31 13:30:00.417143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.416894 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26\" (UID: \"2bcf7019-e8dc-4f70-a37c-3e2640022b8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" Oct 31 13:30:00.417143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.416940 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88drw\" (UniqueName: \"kubernetes.io/projected/4d5a5f23-878d-436d-8f2a-7344d42779dd-kube-api-access-88drw\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2\" (UID: \"4d5a5f23-878d-436d-8f2a-7344d42779dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" Oct 31 13:30:00.417143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.416986 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5a5f23-878d-436d-8f2a-7344d42779dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2\" (UID: \"4d5a5f23-878d-436d-8f2a-7344d42779dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" Oct 31 13:30:00.418050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.418005 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26\" (UID: \"2bcf7019-e8dc-4f70-a37c-3e2640022b8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" Oct 31 13:30:00.418310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.418214 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f66256-f121-47a7-b1a9-f6cb089802c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q\" (UID: \"08f66256-f121-47a7-b1a9-f6cb089802c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" Oct 31 13:30:00.424215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.424102 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2"] Oct 31 13:30:00.432582 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.432518 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlqw5\" (UniqueName: \"kubernetes.io/projected/08f66256-f121-47a7-b1a9-f6cb089802c0-kube-api-access-jlqw5\") pod \"maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q\" (UID: \"08f66256-f121-47a7-b1a9-f6cb089802c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" Oct 31 13:30:00.437318 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.437288 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw"] Oct 31 13:30:00.519155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.519115 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw\" (UID: \"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" Oct 31 13:30:00.519457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.519433 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78p5c\" (UniqueName: \"kubernetes.io/projected/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-kube-api-access-78p5c\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw\" (UID: \"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" Oct 31 13:30:00.519694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.519660 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88drw\" (UniqueName: \"kubernetes.io/projected/4d5a5f23-878d-436d-8f2a-7344d42779dd-kube-api-access-88drw\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2\" (UID: \"4d5a5f23-878d-436d-8f2a-7344d42779dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" Oct 31 13:30:00.519863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.519845 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5a5f23-878d-436d-8f2a-7344d42779dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2\" (UID: \"4d5a5f23-878d-436d-8f2a-7344d42779dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" Oct 31 13:30:00.522441 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.522413 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw\" (UID: \"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" Oct 31 13:30:00.531707 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.524492 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5a5f23-878d-436d-8f2a-7344d42779dd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2\" (UID: \"4d5a5f23-878d-436d-8f2a-7344d42779dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" Oct 31 13:30:00.531911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.526479 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" event={"ID":"7bb8d480-1daf-41eb-aab7-e988dc35c2d6","Type":"ContainerStarted","Data":"cb70eabf0f4a3973b1e09e720ad54412ce4f38b87d9d8fb3770624996a7cdb49"} Oct 31 13:30:00.532064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.532040 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" event={"ID":"7bb8d480-1daf-41eb-aab7-e988dc35c2d6","Type":"ContainerStarted","Data":"76a5752d72c5809913a266b2a25c5cbecc7424ca96e581462015d3383d18f406"} Oct 31 13:30:00.532201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.529905 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l55ff\" (UniqueName: \"kubernetes.io/projected/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-kube-api-access-l55ff\") pod \"maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26\" (UID: \"2bcf7019-e8dc-4f70-a37c-3e2640022b8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" Oct 31 13:30:00.532201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.532179 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" event={"ID":"761abd89-b630-42f8-91bc-6096ef23d6a4","Type":"ContainerStarted","Data":"14a48974c3b3a300219f53a4b7f3fe489c4e899c58f4501dd66ff06f0419e044"} Oct 31 13:30:00.532201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.532203 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" event={"ID":"761abd89-b630-42f8-91bc-6096ef23d6a4","Type":"ContainerStarted","Data":"ce6f5ea5165bc2e7a49d95e234b77738dff2587372c0b72c0bbe5c43686fa108"} Oct 31 13:30:00.534265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.534239 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" event={"ID":"301589df-3ff3-4a56-a4cf-7ca0e1471852","Type":"ContainerDied","Data":"b89f36e08d1632fefea379925b6e3a6997d96f0ed87fe7561e1c800def20d4ed"} Oct 31 13:30:00.534368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.534282 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b89f36e08d1632fefea379925b6e3a6997d96f0ed87fe7561e1c800def20d4ed" Oct 31 13:30:00.534368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.534342 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm" Oct 31 13:30:00.538143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.538110 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78p5c\" (UniqueName: \"kubernetes.io/projected/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-kube-api-access-78p5c\") pod \"maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw\" (UID: \"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" Oct 31 13:30:00.538268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.538121 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88drw\" (UniqueName: \"kubernetes.io/projected/4d5a5f23-878d-436d-8f2a-7344d42779dd-kube-api-access-88drw\") pod \"maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2\" (UID: \"4d5a5f23-878d-436d-8f2a-7344d42779dd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" Oct 31 13:30:00.543338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.543312 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:30:00.582894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.570537 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 13:30:00.607746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.606375 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 13:30:00.615631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.615341 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" Oct 31 13:30:00.616054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.616025 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" Oct 31 13:30:00.705716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.705582 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podStartSLOduration=1.705562394 podStartE2EDuration="1.705562394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:00.454585309 +0000 UTC m=+1568.600329166" watchObservedRunningTime="2026-10-31 13:30:00.705562394 +0000 UTC m=+1568.851306213" Oct 31 13:30:00.723574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.723506 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podStartSLOduration=2.723485558 podStartE2EDuration="2.723485558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:00.570219133 +0000 UTC m=+1568.715962962" watchObservedRunningTime="2026-10-31 13:30:00.723485558 +0000 UTC m=+1568.869229384" Oct 31 13:30:00.738126 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.738094 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz"] Oct 31 13:30:00.778355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.778319 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" Oct 31 13:30:00.789669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.789643 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:30:00.801250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.801150 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz"] Oct 31 13:30:00.803946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.803900 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podStartSLOduration=2.803886475 podStartE2EDuration="2.803886475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:00.629441218 +0000 UTC m=+1568.775185048" watchObservedRunningTime="2026-10-31 13:30:00.803886475 +0000 UTC m=+1568.949630303" Oct 31 13:30:00.824121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.824084 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" Oct 31 13:30:00.826302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.826217 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" Oct 31 13:30:00.850622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.839026 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64c88169-gkhzz\" (UID: \"bbca6d33-4441-4d6c-94cb-b9fa65689083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" Oct 31 13:30:00.850622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.839280 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64c88169-gkhzz\" (UID: \"bbca6d33-4441-4d6c-94cb-b9fa65689083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" Oct 31 13:30:00.850622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.839345 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m45tz\" (UniqueName: \"kubernetes.io/projected/bbca6d33-4441-4d6c-94cb-b9fa65689083-kube-api-access-m45tz\") pod \"maas-api-revoke-sub-64c88169-gkhzz\" (UID: \"bbca6d33-4441-4d6c-94cb-b9fa65689083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" Oct 31 13:30:01.019305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.982179 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m45tz\" (UniqueName: \"kubernetes.io/projected/bbca6d33-4441-4d6c-94cb-b9fa65689083-kube-api-access-m45tz\") pod \"maas-api-revoke-sub-64c88169-gkhzz\" (UID: \"bbca6d33-4441-4d6c-94cb-b9fa65689083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" Oct 31 13:30:01.019305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.982313 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64c88169-gkhzz\" (UID: \"bbca6d33-4441-4d6c-94cb-b9fa65689083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" Oct 31 13:30:01.019305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.982700 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64c88169-gkhzz\" (UID: \"bbca6d33-4441-4d6c-94cb-b9fa65689083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" Oct 31 13:30:01.019305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:00.983583 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64c88169-gkhzz\" (UID: \"bbca6d33-4441-4d6c-94cb-b9fa65689083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" Oct 31 13:30:01.032944 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:01.032086 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64c88169-gkhzz\" (UID: \"bbca6d33-4441-4d6c-94cb-b9fa65689083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" Oct 31 13:30:01.042624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:01.038806 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m45tz\" (UniqueName: \"kubernetes.io/projected/bbca6d33-4441-4d6c-94cb-b9fa65689083-kube-api-access-m45tz\") pod \"maas-api-revoke-sub-64c88169-gkhzz\" (UID: \"bbca6d33-4441-4d6c-94cb-b9fa65689083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" Oct 31 13:30:01.127618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:01.127567 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:30:01.127985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:01.127961 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 13:30:01.128148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:01.128123 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 13:30:01.131201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:01.131177 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" Oct 31 13:30:01.196617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:01.186805 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:30:01.228631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:01.220179 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:30:01.228631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:01.220244 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 13:30:01.882120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:01.879936 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 13:30:01.882120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:01.880008 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 13:30:02.214618 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:02.185860 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:30:02.216656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:02.216525 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:30:02.216894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:02.216864 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 13:30:02.229732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:02.229696 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 13:30:02.229925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:02.229898 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 13:30:02.265616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:02.239152 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q"] Oct 31 13:30:02.327475 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:02.327442 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2"] Oct 31 13:30:02.541820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:02.541368 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 13:30:02.541820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:02.541428 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 13:30:02.629208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:02.628522 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26"] Oct 31 13:30:02.736693 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:02.734703 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw"] Oct 31 13:30:02.745070 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:02.745027 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz"] Oct 31 13:30:03.104148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:03.104110 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" event={"ID":"2bcf7019-e8dc-4f70-a37c-3e2640022b8c","Type":"ContainerStarted","Data":"85bf781cf89dfd910d5f77e107b22408f65ba963f6c87aaa6a635258a92a5c3b"} Oct 31 13:30:03.125985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:03.125846 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 13:30:03.126402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:03.126239 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 13:30:03.166096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:03.162141 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" event={"ID":"08f66256-f121-47a7-b1a9-f6cb089802c0","Type":"ContainerStarted","Data":"52d731b47a81db013bab7c82ee6d1cbccd9b1a61928227b9d34e374b5e76a765"} Oct 31 13:30:03.216745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:03.216708 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:30:03.216955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:03.216927 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 13:30:03.232819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:03.232646 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" event={"ID":"bbca6d33-4441-4d6c-94cb-b9fa65689083","Type":"ContainerStarted","Data":"c7ac0dd2e5c0dd211db8f414630fa37ae5e69c5c9dbb917d7da000f32f6efee6"} Oct 31 13:30:03.237997 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:03.237672 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" event={"ID":"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2","Type":"ContainerStarted","Data":"5338d5a3e0eb4a5594abcc6b8d201c65a077be57238975ec8e7bf32cfbd1fc5c"} Oct 31 13:30:03.327824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:03.321100 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" event={"ID":"4d5a5f23-878d-436d-8f2a-7344d42779dd","Type":"ContainerStarted","Data":"487eb236c09265b206ef55174d0cd58efa87061e8644b253b769430fdfdf13a8"} Oct 31 13:30:03.552644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:03.541425 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 13:30:03.552644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:03.541484 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 13:30:04.122968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.122934 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 13:30:04.123530 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.123498 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 13:30:04.180209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.180174 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:30:04.180472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.180443 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 13:30:04.575621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.574768 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 13:30:04.575621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.574836 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 13:30:04.695254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.676309 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" event={"ID":"bbca6d33-4441-4d6c-94cb-b9fa65689083","Type":"ContainerStarted","Data":"c41b47ac255a29fbc17582cfe760e0de90ab948b2dbd2f6effb89b990261435e"} Oct 31 13:30:04.715123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.708832 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" event={"ID":"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2","Type":"ContainerStarted","Data":"8b61d54fc07189147b0b38fa5f9e26686b0b8e0d96559d9aab65aa3fe2dfdf48"} Oct 31 13:30:04.715123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.713975 5532 generic.go:356] "Generic (PLEG): container finished" podID="4d5a5f23-878d-436d-8f2a-7344d42779dd" containerID="b71a14627373c6e396f2d133004bef29a28af7f1cb23dad35fe08efb7e6da6ef" exitCode=60 Oct 31 13:30:04.715123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.714106 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" event={"ID":"4d5a5f23-878d-436d-8f2a-7344d42779dd","Type":"ContainerDied","Data":"b71a14627373c6e396f2d133004bef29a28af7f1cb23dad35fe08efb7e6da6ef"} Oct 31 13:30:04.715123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.714583 5532 scope.go:122] "RemoveContainer" containerID="b71a14627373c6e396f2d133004bef29a28af7f1cb23dad35fe08efb7e6da6ef" Oct 31 13:30:04.718640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.716398 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" podStartSLOduration=4.716380653 podStartE2EDuration="4.716380653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:04.712761992 +0000 UTC m=+1572.858505821" watchObservedRunningTime="2026-10-31 13:30:04.716380653 +0000 UTC m=+1572.862124482" Oct 31 13:30:04.814216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.778052 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" event={"ID":"2bcf7019-e8dc-4f70-a37c-3e2640022b8c","Type":"ContainerStarted","Data":"58223c1e12a2edf84fdcbd08fc666cef78d06f87a6fb0da4115d400bfa09aeea"} Oct 31 13:30:04.814216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.778673 5532 scope.go:122] "RemoveContainer" containerID="58223c1e12a2edf84fdcbd08fc666cef78d06f87a6fb0da4115d400bfa09aeea" Oct 31 13:30:04.860675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.856452 5532 generic.go:356] "Generic (PLEG): container finished" podID="08f66256-f121-47a7-b1a9-f6cb089802c0" containerID="6c82f35cd4ce2b8c51c2c1740cfde2a8ca702d82d7d2b343ac84872a4c124dbd" exitCode=60 Oct 31 13:30:04.860675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.859098 5532 scope.go:122] "RemoveContainer" containerID="6c82f35cd4ce2b8c51c2c1740cfde2a8ca702d82d7d2b343ac84872a4c124dbd" Oct 31 13:30:04.900651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.898647 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" event={"ID":"08f66256-f121-47a7-b1a9-f6cb089802c0","Type":"ContainerDied","Data":"6c82f35cd4ce2b8c51c2c1740cfde2a8ca702d82d7d2b343ac84872a4c124dbd"} Oct 31 13:30:05.003838 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:04.965213 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" podStartSLOduration=4.965189178 podStartE2EDuration="4.965189178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:04.938544489 +0000 UTC m=+1573.084288325" watchObservedRunningTime="2026-10-31 13:30:04.965189178 +0000 UTC m=+1573.110933007" Oct 31 13:30:05.232777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:05.232741 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 13:30:05.233294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:05.233261 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 13:30:05.262620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:05.235062 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:30:05.295828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:05.281036 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 13:30:05.423621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:05.423378 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded" Oct 31 13:30:05.439912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:05.437732 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:05.439912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:05.438161 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:30:05.586610 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:05.551723 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 13:30:05.586610 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:05.551797 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 13:30:05.586610 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:05.585691 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:05.618913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:05.617686 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:05.780697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:05.680700 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:05.780697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:05.720015 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:06.078152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.075846 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.172779 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.172846 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.288139 5532 generic.go:356] "Generic (PLEG): container finished" podID="bbca6d33-4441-4d6c-94cb-b9fa65689083" containerID="c41b47ac255a29fbc17582cfe760e0de90ab948b2dbd2f6effb89b990261435e" exitCode=0 Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.311802 5532 generic.go:356] "Generic (PLEG): container finished" podID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" containerID="8b61d54fc07189147b0b38fa5f9e26686b0b8e0d96559d9aab65aa3fe2dfdf48" exitCode=60 Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.312550 5532 scope.go:122] "RemoveContainer" containerID="8b61d54fc07189147b0b38fa5f9e26686b0b8e0d96559d9aab65aa3fe2dfdf48" Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.451726 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.451779 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.480189 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.523749 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.583873 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.627747 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.627802 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.628428 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.628684 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.640710 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.640775 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" podUID="dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.641373 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:30:06.678721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.655675 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:06.837628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.683975 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:06.837628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.772047 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" event={"ID":"bbca6d33-4441-4d6c-94cb-b9fa65689083","Type":"ContainerDied","Data":"c41b47ac255a29fbc17582cfe760e0de90ab948b2dbd2f6effb89b990261435e"} Oct 31 13:30:06.837628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.772119 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" event={"ID":"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2","Type":"ContainerDied","Data":"8b61d54fc07189147b0b38fa5f9e26686b0b8e0d96559d9aab65aa3fe2dfdf48"} Oct 31 13:30:06.837628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.772194 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:30:06.837628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.772273 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:30:06.837628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.772313 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:30:06.837628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.772350 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:30:06.837628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.772386 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:30:06.964815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.920143 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:30:06.964815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.926730 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:30:06.964815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.927211 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:06.964815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:06.952688 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:07.090949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.078694 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:30:07.115686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.103227 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:30:07.147201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.147077 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 13:30:07.147201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.147140 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 13:30:07.182620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.171885 5532 generic.go:356] "Generic (PLEG): container finished" podID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" containerID="58223c1e12a2edf84fdcbd08fc666cef78d06f87a6fb0da4115d400bfa09aeea" exitCode=60 Oct 31 13:30:07.182620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.172089 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" event={"ID":"2bcf7019-e8dc-4f70-a37c-3e2640022b8c","Type":"ContainerDied","Data":"58223c1e12a2edf84fdcbd08fc666cef78d06f87a6fb0da4115d400bfa09aeea"} Oct 31 13:30:07.188355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.187744 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:30:07.188355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.187810 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 13:30:07.521754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.521700 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:07.604254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.604212 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:30:07.713445 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.713394 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:30:07.813620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.769222 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:30:07.813620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.770042 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 31 13:30:07.817970 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.817923 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": context deadline exceeded" Oct 31 13:30:07.823627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.818339 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 13:30:07.823903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.823864 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 13:30:07.850716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.849208 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:07.993621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:07.942701 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:30:08.179883 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.050313 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:30:08.179883 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.175817 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:30:08.343621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.329929 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:30:08.343621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.330004 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 13:30:08.424655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.406167 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:30:08.522110 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.050442 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:30:08.603621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.591028 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 13:30:08.603621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.591111 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 13:30:08.603621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.591426 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:30:08.603621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.592233 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" podUID="0b01eeee-1f42-4e8e-af23-4e228345c60f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:08.603621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.600250 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 13:30:08.618195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.618150 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:08.659008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.658938 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:30:08.660228 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.660180 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" podUID="5a9332ef-beec-4b46-9617-36d8ce8812c5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:08.716878 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.603844 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 13:30:08.717468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.717428 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:30:08.792681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.755696 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:08.793034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.792976 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:09.010173 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.010123 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:30:09.066679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:08.857084 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:09.132200 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.081646 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:30:09.195622 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:09.152152 5532 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.077s" Oct 31 13:30:09.260621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.230817 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:30:09.261079 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.261050 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:30:09.295804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.288226 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:30:09.295804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.284942 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:30:09.295804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.288795 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 13:30:09.295804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.180714 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:09.295804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.180779 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:09.295804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.285029 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 13:30:09.295804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.291719 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 13:30:09.353766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.331729 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:30:09.353766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.331795 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:09.353766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.343324 5532 generic.go:356] "Generic (PLEG): container finished" podID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" containerID="e1b20969df73f772811e64431a978f471f67b40ceb3665e2c53b3316576cabbc" exitCode=60 Oct 31 13:30:09.353766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.343427 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" event={"ID":"2bcf7019-e8dc-4f70-a37c-3e2640022b8c","Type":"ContainerDied","Data":"e1b20969df73f772811e64431a978f471f67b40ceb3665e2c53b3316576cabbc"} Oct 31 13:30:09.353766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.343488 5532 scope.go:122] "RemoveContainer" containerID="58223c1e12a2edf84fdcbd08fc666cef78d06f87a6fb0da4115d400bfa09aeea" Oct 31 13:30:09.353766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.344095 5532 scope.go:122] "RemoveContainer" containerID="e1b20969df73f772811e64431a978f471f67b40ceb3665e2c53b3316576cabbc" Oct 31 13:30:09.353766 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:09.344588 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26_odh-ai-gateway-infra(2bcf7019-e8dc-4f70-a37c-3e2640022b8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" podUID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" Oct 31 13:30:09.374062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.368240 5532 generic.go:356] "Generic (PLEG): container finished" podID="08f66256-f121-47a7-b1a9-f6cb089802c0" containerID="0395941bda223954b477f7ffb75f10b43e65f5486ff997b52b3c1ef07e1ab3f6" exitCode=60 Oct 31 13:30:09.374062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.368557 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" event={"ID":"08f66256-f121-47a7-b1a9-f6cb089802c0","Type":"ContainerDied","Data":"0395941bda223954b477f7ffb75f10b43e65f5486ff997b52b3c1ef07e1ab3f6"} Oct 31 13:30:09.375686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.375136 5532 scope.go:122] "RemoveContainer" containerID="0395941bda223954b477f7ffb75f10b43e65f5486ff997b52b3c1ef07e1ab3f6" Oct 31 13:30:09.375686 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:09.375618 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q_odh-ai-gateway-infra(08f66256-f121-47a7-b1a9-f6cb089802c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" podUID="08f66256-f121-47a7-b1a9-f6cb089802c0" Oct 31 13:30:09.425656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.408302 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:09.541621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.517615 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:09.555073 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.553751 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 13:30:09.555073 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.553818 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 13:30:09.610359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.610258 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:09.672621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.648668 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:09.752621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.722717 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:09.793945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.793900 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" event={"ID":"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2","Type":"ContainerStarted","Data":"fb4cf1b312be833a0474b92782f4b2065a8b715b838b84caab8993d5bae0fd12"} Oct 31 13:30:09.823702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.823644 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:09.824222 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.824144 5532 scope.go:122] "RemoveContainer" containerID="fb4cf1b312be833a0474b92782f4b2065a8b715b838b84caab8993d5bae0fd12" Oct 31 13:30:09.848629 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:09.824717 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw_odh-ai-gateway-infra(cd813f2d-c7d1-4ad5-937f-2f78d6daebc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" Oct 31 13:30:09.892648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:09.888618 5532 scope.go:122] "RemoveContainer" containerID="6c82f35cd4ce2b8c51c2c1740cfde2a8ca702d82d7d2b343ac84872a4c124dbd" Oct 31 13:30:10.049140 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.049092 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:10.076621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.060729 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:10.090237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.090183 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:10.118753 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.118712 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:30:10.119053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.119019 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:10.156617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.152673 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:10.189622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.185712 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:30:10.320631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.247776 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 13:30:10.320631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.206213 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 13:30:10.320631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.247854 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 13:30:10.320631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.309911 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:10.320631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.310345 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:30:10.404679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.385615 5532 generic.go:356] "Generic (PLEG): container finished" podID="4d5a5f23-878d-436d-8f2a-7344d42779dd" containerID="211c8190965d64c9600d8a7772d540f9eb8431012d78320756c55ec8a32b1f57" exitCode=60 Oct 31 13:30:10.421717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.417736 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:30:10.421717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.417795 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:10.421717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.419536 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" event={"ID":"4d5a5f23-878d-436d-8f2a-7344d42779dd","Type":"ContainerDied","Data":"211c8190965d64c9600d8a7772d540f9eb8431012d78320756c55ec8a32b1f57"} Oct 31 13:30:10.421717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.419656 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:10.596639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.573836 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:10.626319 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.622293 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 13:30:10.626319 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.622361 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 13:30:10.674004 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.646081 5532 scope.go:122] "RemoveContainer" containerID="211c8190965d64c9600d8a7772d540f9eb8431012d78320756c55ec8a32b1f57" Oct 31 13:30:10.674004 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:10.646484 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2_odh-ai-gateway-infra(4d5a5f23-878d-436d-8f2a-7344d42779dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" podUID="4d5a5f23-878d-436d-8f2a-7344d42779dd" Oct 31 13:30:10.674004 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.647088 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:10.674004 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.673444 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:30:10.732630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.730607 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:10.851622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.848254 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:11.000117 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:10.999928 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:11.068917 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.068872 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:11.070010 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.069972 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:11.117163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.117114 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:11.160799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.160761 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body= Oct 31 13:30:11.161515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.161483 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" Oct 31 13:30:11.180518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.180482 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Oct 31 13:30:11.180771 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.180742 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Oct 31 13:30:11.314835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.314724 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:11.316863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.316821 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:30:11.419135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.406690 5532 scope.go:122] "RemoveContainer" containerID="b71a14627373c6e396f2d133004bef29a28af7f1cb23dad35fe08efb7e6da6ef" Oct 31 13:30:11.425682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.425630 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:30:11.561323 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.545077 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 13:30:11.768865 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.768822 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 13:30:12.329786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.669213 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:30:12.329786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.664945 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:30:12.329786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.696434 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:30:12.329786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.699255 5532 generic.go:356] "Generic (PLEG): container finished" podID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" containerID="fb4cf1b312be833a0474b92782f4b2065a8b715b838b84caab8993d5bae0fd12" exitCode=60 Oct 31 13:30:12.329786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.704496 5532 scope.go:122] "RemoveContainer" containerID="fb4cf1b312be833a0474b92782f4b2065a8b715b838b84caab8993d5bae0fd12" Oct 31 13:30:12.329786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.754669 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:12.329786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.755475 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" podUID="51b75119-a5cd-4e8c-bd06-3725a6bb30a5" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.29:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:12.329786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:11.767343 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" event={"ID":"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2","Type":"ContainerDied","Data":"fb4cf1b312be833a0474b92782f4b2065a8b715b838b84caab8993d5bae0fd12"} Oct 31 13:30:12.329786 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:12.013301 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw_odh-ai-gateway-infra(cd813f2d-c7d1-4ad5-937f-2f78d6daebc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" Oct 31 13:30:12.329786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.178705 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:12.329786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.253030 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:30:12.363727 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.362482 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:30:12.373641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.368451 5532 scope.go:122] "RemoveContainer" containerID="8b61d54fc07189147b0b38fa5f9e26686b0b8e0d96559d9aab65aa3fe2dfdf48" Oct 31 13:30:12.430058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.429762 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:30:12.681180 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.679252 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Oct 31 13:30:12.681180 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.679320 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Oct 31 13:30:12.684463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.681894 5532 scope.go:122] "RemoveContainer" containerID="e1b20969df73f772811e64431a978f471f67b40ceb3665e2c53b3316576cabbc" Oct 31 13:30:12.688629 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:12.684761 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26_odh-ai-gateway-infra(2bcf7019-e8dc-4f70-a37c-3e2640022b8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" podUID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" Oct 31 13:30:12.694552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.694517 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" Oct 31 13:30:12.695535 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.695044 5532 scope.go:122] "RemoveContainer" containerID="211c8190965d64c9600d8a7772d540f9eb8431012d78320756c55ec8a32b1f57" Oct 31 13:30:12.695535 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:12.695469 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2_odh-ai-gateway-infra(4d5a5f23-878d-436d-8f2a-7344d42779dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" podUID="4d5a5f23-878d-436d-8f2a-7344d42779dd" Oct 31 13:30:12.703633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.695734 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:30:12.703633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.695786 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:30:12.703633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.695821 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:30:12.703633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.695855 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:30:12.703633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.695887 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:30:12.703633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.695921 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:30:12.703633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.695954 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:30:12.703633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.698691 5532 scope.go:122] "RemoveContainer" containerID="0395941bda223954b477f7ffb75f10b43e65f5486ff997b52b3c1ef07e1ab3f6" Oct 31 13:30:12.703633 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:12.700902 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q_odh-ai-gateway-infra(08f66256-f121-47a7-b1a9-f6cb089802c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" podUID="08f66256-f121-47a7-b1a9-f6cb089802c0" Oct 31 13:30:12.718502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.714520 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:30:12.718502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.714627 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:30:12.718502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.716227 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:30:12.718502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.717074 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:30:12.718502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.718177 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:30:12.733638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.721466 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:30:12.733638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.721521 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:30:12.814736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.814459 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-xks-ca\") pod \"bbca6d33-4441-4d6c-94cb-b9fa65689083\" (UID: \"bbca6d33-4441-4d6c-94cb-b9fa65689083\") " Oct 31 13:30:12.815836 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.815814 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-openshift-ca\") pod \"bbca6d33-4441-4d6c-94cb-b9fa65689083\" (UID: \"bbca6d33-4441-4d6c-94cb-b9fa65689083\") " Oct 31 13:30:12.816737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.816297 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-openshift-ca" pod "bbca6d33-4441-4d6c-94cb-b9fa65689083" (UID: "bbca6d33-4441-4d6c-94cb-b9fa65689083"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:12.818001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.817971 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbca6d33-4441-4d6c-94cb-b9fa65689083-kube-api-access-m45tz\" (UniqueName: \"kubernetes.io/projected/bbca6d33-4441-4d6c-94cb-b9fa65689083-kube-api-access-m45tz\") pod \"bbca6d33-4441-4d6c-94cb-b9fa65689083\" (UID: \"bbca6d33-4441-4d6c-94cb-b9fa65689083\") " Oct 31 13:30:12.819667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.819639 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-xks-ca" pod "bbca6d33-4441-4d6c-94cb-b9fa65689083" (UID: "bbca6d33-4441-4d6c-94cb-b9fa65689083"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:30:12.821779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.821757 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:12.830866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.830838 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbca6d33-4441-4d6c-94cb-b9fa65689083-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:12.831000 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.830016 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbca6d33-4441-4d6c-94cb-b9fa65689083-kube-api-access-m45tz" pod "bbca6d33-4441-4d6c-94cb-b9fa65689083" (UID: "bbca6d33-4441-4d6c-94cb-b9fa65689083"). InnerVolumeSpecName "kube-api-access-m45tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:12.932747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.932652 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m45tz\" (UniqueName: \"kubernetes.io/projected/bbca6d33-4441-4d6c-94cb-b9fa65689083-kube-api-access-m45tz\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:12.957299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.957267 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:30:12.958299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.958249 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:30:12.961029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:12.960988 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:30:13.591665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:13.591626 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:30:13.592367 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:13.592346 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:30:13.593336 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:13.593313 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:30:13.710734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:13.710673 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" event={"ID":"bbca6d33-4441-4d6c-94cb-b9fa65689083","Type":"ContainerDied","Data":"c7ac0dd2e5c0dd211db8f414630fa37ae5e69c5c9dbb917d7da000f32f6efee6"} Oct 31 13:30:13.710734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:13.710713 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz" Oct 31 13:30:13.710980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:13.710779 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7ac0dd2e5c0dd211db8f414630fa37ae5e69c5c9dbb917d7da000f32f6efee6" Oct 31 13:30:13.713890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:13.713857 5532 scope.go:122] "RemoveContainer" containerID="fb4cf1b312be833a0474b92782f4b2065a8b715b838b84caab8993d5bae0fd12" Oct 31 13:30:13.714317 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:13.714287 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw_odh-ai-gateway-infra(cd813f2d-c7d1-4ad5-937f-2f78d6daebc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" Oct 31 13:30:14.564209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.564173 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d"] Oct 31 13:30:14.564913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.564893 5532 cpu_manager.go:414] "containerMap: removing container" podUID="bbca6d33-4441-4d6c-94cb-b9fa65689083" containerName="revoke-keys" Oct 31 13:30:14.564913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.564909 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbca6d33-4441-4d6c-94cb-b9fa65689083" containerName="revoke-keys" Oct 31 13:30:14.565129 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.565046 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbca6d33-4441-4d6c-94cb-b9fa65689083" containerName="revoke-keys" Oct 31 13:30:14.570363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.570335 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" Oct 31 13:30:14.584740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.584706 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:30:14.590689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.589410 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d"] Oct 31 13:30:14.682616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.681577 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74eeca55-qqz9d\" (UID: \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" Oct 31 13:30:14.682616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.681814 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74eeca55-qqz9d\" (UID: \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" Oct 31 13:30:14.682616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.682072 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkcv8\" (UniqueName: \"kubernetes.io/projected/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-kube-api-access-rkcv8\") pod \"maas-api-revoke-sub-74eeca55-qqz9d\" (UID: \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" Oct 31 13:30:14.783973 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.783883 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74eeca55-qqz9d\" (UID: \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" Oct 31 13:30:14.784681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.784233 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkcv8\" (UniqueName: \"kubernetes.io/projected/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-kube-api-access-rkcv8\") pod \"maas-api-revoke-sub-74eeca55-qqz9d\" (UID: \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" Oct 31 13:30:14.784681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.784363 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74eeca55-qqz9d\" (UID: \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" Oct 31 13:30:14.790978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.788124 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74eeca55-qqz9d\" (UID: \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" Oct 31 13:30:14.794302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.794273 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74eeca55-qqz9d\" (UID: \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" Oct 31 13:30:14.808232 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.808202 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkcv8\" (UniqueName: \"kubernetes.io/projected/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-kube-api-access-rkcv8\") pod \"maas-api-revoke-sub-74eeca55-qqz9d\" (UID: \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" Oct 31 13:30:14.915799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:14.915750 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" Oct 31 13:30:15.218216 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:30:15.217338 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa6d4f64_a203_4e4f_9a12_b9fdd4bfe31e.slice/crio-734683d60e88ff0fb222a13b7d80e0ffd4f6c1e9d948c61a44f08db210f0d43b WatchSource:0}: Error finding container 734683d60e88ff0fb222a13b7d80e0ffd4f6c1e9d948c61a44f08db210f0d43b: Status 404 returned error can't find the container with id 734683d60e88ff0fb222a13b7d80e0ffd4f6c1e9d948c61a44f08db210f0d43b Oct 31 13:30:15.220452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:15.220418 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d"] Oct 31 13:30:15.734893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:15.734803 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" event={"ID":"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e","Type":"ContainerStarted","Data":"2ad008cd68eff71450a8e5b7e1f60c0e603e443efa6f6f1db2d9b63e06c44fb4"} Oct 31 13:30:15.734893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:15.734857 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" event={"ID":"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e","Type":"ContainerStarted","Data":"734683d60e88ff0fb222a13b7d80e0ffd4f6c1e9d948c61a44f08db210f0d43b"} Oct 31 13:30:15.766101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:15.766046 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" podStartSLOduration=1.7660303389999998 podStartE2EDuration="1.766030339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:15.76579961 +0000 UTC m=+1583.911543438" watchObservedRunningTime="2026-10-31 13:30:15.766030339 +0000 UTC m=+1583.911774166" Oct 31 13:30:16.762077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:16.761891 5532 generic.go:356] "Generic (PLEG): container finished" podID="fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e" containerID="2ad008cd68eff71450a8e5b7e1f60c0e603e443efa6f6f1db2d9b63e06c44fb4" exitCode=0 Oct 31 13:30:16.762473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:16.762249 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" event={"ID":"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e","Type":"ContainerDied","Data":"2ad008cd68eff71450a8e5b7e1f60c0e603e443efa6f6f1db2d9b63e06c44fb4"} Oct 31 13:30:17.995287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:17.995257 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" Oct 31 13:30:18.044025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:18.043925 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-openshift-ca\") pod \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\" (UID: \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\") " Oct 31 13:30:18.044025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:18.044013 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-xks-ca\") pod \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\" (UID: \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\") " Oct 31 13:30:18.044269 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:18.044156 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-kube-api-access-rkcv8\" (UniqueName: \"kubernetes.io/projected/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-kube-api-access-rkcv8\") pod \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\" (UID: \"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e\") " Oct 31 13:30:18.047779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:18.047679 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-openshift-ca" pod "fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e" (UID: "fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:18.072315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:18.072274 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-kube-api-access-rkcv8" pod "fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e" (UID: "fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e"). InnerVolumeSpecName "kube-api-access-rkcv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:18.072694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:18.072559 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-xks-ca" pod "fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e" (UID: "fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:30:18.146225 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:18.146177 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkcv8\" (UniqueName: \"kubernetes.io/projected/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-kube-api-access-rkcv8\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:18.146225 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:18.146222 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:18.146447 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:18.146235 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:18.781657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:18.781628 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" Oct 31 13:30:18.781875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:18.781615 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d" event={"ID":"fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e","Type":"ContainerDied","Data":"734683d60e88ff0fb222a13b7d80e0ffd4f6c1e9d948c61a44f08db210f0d43b"} Oct 31 13:30:18.781875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:18.781802 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="734683d60e88ff0fb222a13b7d80e0ffd4f6c1e9d948c61a44f08db210f0d43b" Oct 31 13:30:18.996589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:18.996549 5532 scope.go:122] "RemoveContainer" containerID="211c8190965d64c9600d8a7772d540f9eb8431012d78320756c55ec8a32b1f57" Oct 31 13:30:19.549330 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.549295 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8"] Oct 31 13:30:19.550129 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.550104 5532 cpu_manager.go:414] "containerMap: removing container" podUID="fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e" containerName="revoke-keys" Oct 31 13:30:19.550129 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.550126 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e" containerName="revoke-keys" Oct 31 13:30:19.550319 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.550305 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e" containerName="revoke-keys" Oct 31 13:30:19.554300 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.554277 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" Oct 31 13:30:19.563484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.563451 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8"] Oct 31 13:30:19.564649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.564119 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:30:19.673492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.673453 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9799050e-794b-4634-9838-503737ce1cd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f986406f-t6rj8\" (UID: \"9799050e-794b-4634-9838-503737ce1cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" Oct 31 13:30:19.673826 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.673798 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9799050e-794b-4634-9838-503737ce1cd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f986406f-t6rj8\" (UID: \"9799050e-794b-4634-9838-503737ce1cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" Oct 31 13:30:19.673958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.673938 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bcm2\" (UniqueName: \"kubernetes.io/projected/9799050e-794b-4634-9838-503737ce1cd0-kube-api-access-2bcm2\") pod \"maas-api-revoke-sub-f986406f-t6rj8\" (UID: \"9799050e-794b-4634-9838-503737ce1cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" Oct 31 13:30:19.775868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.775836 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9799050e-794b-4634-9838-503737ce1cd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f986406f-t6rj8\" (UID: \"9799050e-794b-4634-9838-503737ce1cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" Oct 31 13:30:19.776047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.775940 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bcm2\" (UniqueName: \"kubernetes.io/projected/9799050e-794b-4634-9838-503737ce1cd0-kube-api-access-2bcm2\") pod \"maas-api-revoke-sub-f986406f-t6rj8\" (UID: \"9799050e-794b-4634-9838-503737ce1cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" Oct 31 13:30:19.776047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.776035 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9799050e-794b-4634-9838-503737ce1cd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f986406f-t6rj8\" (UID: \"9799050e-794b-4634-9838-503737ce1cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" Oct 31 13:30:19.776723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.776693 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9799050e-794b-4634-9838-503737ce1cd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f986406f-t6rj8\" (UID: \"9799050e-794b-4634-9838-503737ce1cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" Oct 31 13:30:19.778434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.778410 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9799050e-794b-4634-9838-503737ce1cd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f986406f-t6rj8\" (UID: \"9799050e-794b-4634-9838-503737ce1cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" Oct 31 13:30:19.792559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.792530 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bcm2\" (UniqueName: \"kubernetes.io/projected/9799050e-794b-4634-9838-503737ce1cd0-kube-api-access-2bcm2\") pod \"maas-api-revoke-sub-f986406f-t6rj8\" (UID: \"9799050e-794b-4634-9838-503737ce1cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" Oct 31 13:30:19.795993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.795968 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" event={"ID":"4d5a5f23-878d-436d-8f2a-7344d42779dd","Type":"ContainerStarted","Data":"20cbbc9f3d3752009f9141c59e03d7a9cca47cfffc0b327e9d0afddc74a31d5f"} Oct 31 13:30:19.820998 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.820950 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" podStartSLOduration=19.820935392 podStartE2EDuration="19.820935392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:19.818041505 +0000 UTC m=+1587.963785333" watchObservedRunningTime="2026-10-31 13:30:19.820935392 +0000 UTC m=+1587.966679220" Oct 31 13:30:19.884385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.884350 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" Oct 31 13:30:19.999490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:19.999458 5532 scope.go:122] "RemoveContainer" containerID="e1b20969df73f772811e64431a978f471f67b40ceb3665e2c53b3316576cabbc" Oct 31 13:30:20.002185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.002163 5532 scope.go:122] "RemoveContainer" containerID="fb4cf1b312be833a0474b92782f4b2065a8b715b838b84caab8993d5bae0fd12" Oct 31 13:30:20.004773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.004753 5532 scope.go:122] "RemoveContainer" containerID="0395941bda223954b477f7ffb75f10b43e65f5486ff997b52b3c1ef07e1ab3f6" Oct 31 13:30:20.177136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.177107 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8"] Oct 31 13:30:20.177397 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:30:20.177367 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9799050e_794b_4634_9838_503737ce1cd0.slice/crio-c7db0532506b33c6c3e47417df1b740ab95016cded0f28a69b4b4dcf76db6431 WatchSource:0}: Error finding container c7db0532506b33c6c3e47417df1b740ab95016cded0f28a69b4b4dcf76db6431: Status 404 returned error can't find the container with id c7db0532506b33c6c3e47417df1b740ab95016cded0f28a69b4b4dcf76db6431 Oct 31 13:30:20.806401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.806292 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" event={"ID":"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2","Type":"ContainerStarted","Data":"b3190e2445eb5570c569e49bb2f20409cc1627a440523ec192a3d3b77511869e"} Oct 31 13:30:20.808312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.808282 5532 generic.go:356] "Generic (PLEG): container finished" podID="4d5a5f23-878d-436d-8f2a-7344d42779dd" containerID="20cbbc9f3d3752009f9141c59e03d7a9cca47cfffc0b327e9d0afddc74a31d5f" exitCode=60 Oct 31 13:30:20.808465 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.808360 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" event={"ID":"4d5a5f23-878d-436d-8f2a-7344d42779dd","Type":"ContainerDied","Data":"20cbbc9f3d3752009f9141c59e03d7a9cca47cfffc0b327e9d0afddc74a31d5f"} Oct 31 13:30:20.808465 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.808421 5532 scope.go:122] "RemoveContainer" containerID="211c8190965d64c9600d8a7772d540f9eb8431012d78320756c55ec8a32b1f57" Oct 31 13:30:20.808793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.808774 5532 scope.go:122] "RemoveContainer" containerID="20cbbc9f3d3752009f9141c59e03d7a9cca47cfffc0b327e9d0afddc74a31d5f" Oct 31 13:30:20.809176 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:20.809150 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2_odh-ai-gateway-infra(4d5a5f23-878d-436d-8f2a-7344d42779dd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" podUID="4d5a5f23-878d-436d-8f2a-7344d42779dd" Oct 31 13:30:20.810565 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.810539 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" event={"ID":"2bcf7019-e8dc-4f70-a37c-3e2640022b8c","Type":"ContainerStarted","Data":"d8bc8df7cc1ec016365f934cded8af026188e4d13c1b2f38f619f05c74d30586"} Oct 31 13:30:20.812519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.812499 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" event={"ID":"08f66256-f121-47a7-b1a9-f6cb089802c0","Type":"ContainerStarted","Data":"ba2926805b3c95f7992e3cb2a9c5044e30d1f5f09ddf5c9e2f762a2efff85686"} Oct 31 13:30:20.814543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.814253 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" event={"ID":"9799050e-794b-4634-9838-503737ce1cd0","Type":"ContainerStarted","Data":"54e42553dc1b95a505ec7df6030d5c7b4a9fc04bb60ba7d95c4077fe5b338df0"} Oct 31 13:30:20.814543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.814302 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" event={"ID":"9799050e-794b-4634-9838-503737ce1cd0","Type":"ContainerStarted","Data":"c7db0532506b33c6c3e47417df1b740ab95016cded0f28a69b4b4dcf76db6431"} Oct 31 13:30:20.833770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.833748 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2"] Oct 31 13:30:20.860065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.860017 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" podStartSLOduration=20.860004654 podStartE2EDuration="20.860004654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:20.857300782 +0000 UTC m=+1589.003044610" watchObservedRunningTime="2026-10-31 13:30:20.860004654 +0000 UTC m=+1589.005748482" Oct 31 13:30:20.891752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.891697 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" podStartSLOduration=1.891684613 podStartE2EDuration="1.891684613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:20.887403514 +0000 UTC m=+1589.033147344" watchObservedRunningTime="2026-10-31 13:30:20.891684613 +0000 UTC m=+1589.037428441" Oct 31 13:30:20.918142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:20.918089 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" podStartSLOduration=20.918076249 podStartE2EDuration="20.918076249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:20.915272258 +0000 UTC m=+1589.061016092" watchObservedRunningTime="2026-10-31 13:30:20.918076249 +0000 UTC m=+1589.063820076" Oct 31 13:30:21.826691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.826654 5532 generic.go:356] "Generic (PLEG): container finished" podID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" containerID="b3190e2445eb5570c569e49bb2f20409cc1627a440523ec192a3d3b77511869e" exitCode=60 Oct 31 13:30:21.827182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.826740 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" event={"ID":"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2","Type":"ContainerDied","Data":"b3190e2445eb5570c569e49bb2f20409cc1627a440523ec192a3d3b77511869e"} Oct 31 13:30:21.827182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.826807 5532 scope.go:122] "RemoveContainer" containerID="fb4cf1b312be833a0474b92782f4b2065a8b715b838b84caab8993d5bae0fd12" Oct 31 13:30:21.827266 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.827242 5532 scope.go:122] "RemoveContainer" containerID="b3190e2445eb5570c569e49bb2f20409cc1627a440523ec192a3d3b77511869e" Oct 31 13:30:21.827683 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:21.827656 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw_odh-ai-gateway-infra(cd813f2d-c7d1-4ad5-937f-2f78d6daebc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" Oct 31 13:30:21.831756 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.831733 5532 generic.go:356] "Generic (PLEG): container finished" podID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" containerID="d8bc8df7cc1ec016365f934cded8af026188e4d13c1b2f38f619f05c74d30586" exitCode=60 Oct 31 13:30:21.831880 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.831807 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" event={"ID":"2bcf7019-e8dc-4f70-a37c-3e2640022b8c","Type":"ContainerDied","Data":"d8bc8df7cc1ec016365f934cded8af026188e4d13c1b2f38f619f05c74d30586"} Oct 31 13:30:21.832489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.832463 5532 scope.go:122] "RemoveContainer" containerID="d8bc8df7cc1ec016365f934cded8af026188e4d13c1b2f38f619f05c74d30586" Oct 31 13:30:21.832921 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:21.832897 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26_odh-ai-gateway-infra(2bcf7019-e8dc-4f70-a37c-3e2640022b8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" podUID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" Oct 31 13:30:21.834194 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.834175 5532 generic.go:356] "Generic (PLEG): container finished" podID="08f66256-f121-47a7-b1a9-f6cb089802c0" containerID="ba2926805b3c95f7992e3cb2a9c5044e30d1f5f09ddf5c9e2f762a2efff85686" exitCode=60 Oct 31 13:30:21.834299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.834249 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" event={"ID":"08f66256-f121-47a7-b1a9-f6cb089802c0","Type":"ContainerDied","Data":"ba2926805b3c95f7992e3cb2a9c5044e30d1f5f09ddf5c9e2f762a2efff85686"} Oct 31 13:30:21.834732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.834713 5532 scope.go:122] "RemoveContainer" containerID="ba2926805b3c95f7992e3cb2a9c5044e30d1f5f09ddf5c9e2f762a2efff85686" Oct 31 13:30:21.835098 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:21.835074 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q_odh-ai-gateway-infra(08f66256-f121-47a7-b1a9-f6cb089802c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" podUID="08f66256-f121-47a7-b1a9-f6cb089802c0" Oct 31 13:30:21.836336 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.836316 5532 generic.go:356] "Generic (PLEG): container finished" podID="9799050e-794b-4634-9838-503737ce1cd0" containerID="54e42553dc1b95a505ec7df6030d5c7b4a9fc04bb60ba7d95c4077fe5b338df0" exitCode=0 Oct 31 13:30:21.836418 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.836367 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" event={"ID":"9799050e-794b-4634-9838-503737ce1cd0","Type":"ContainerDied","Data":"54e42553dc1b95a505ec7df6030d5c7b4a9fc04bb60ba7d95c4077fe5b338df0"} Oct 31 13:30:21.842915 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.842891 5532 scope.go:122] "RemoveContainer" containerID="e1b20969df73f772811e64431a978f471f67b40ceb3665e2c53b3316576cabbc" Oct 31 13:30:21.846922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.846896 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw"] Oct 31 13:30:21.857038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.857018 5532 scope.go:122] "RemoveContainer" containerID="0395941bda223954b477f7ffb75f10b43e65f5486ff997b52b3c1ef07e1ab3f6" Oct 31 13:30:21.877040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.877013 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q"] Oct 31 13:30:21.938172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:21.938140 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26"] Oct 31 13:30:22.041789 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:22.041764 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" Oct 31 13:30:22.222981 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:22.222942 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d5a5f23-878d-436d-8f2a-7344d42779dd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5a5f23-878d-436d-8f2a-7344d42779dd-maas-api-service-ca\") pod \"4d5a5f23-878d-436d-8f2a-7344d42779dd\" (UID: \"4d5a5f23-878d-436d-8f2a-7344d42779dd\") " Oct 31 13:30:22.223166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:22.223045 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5a5f23-878d-436d-8f2a-7344d42779dd-kube-api-access-88drw\" (UniqueName: \"kubernetes.io/projected/4d5a5f23-878d-436d-8f2a-7344d42779dd-kube-api-access-88drw\") pod \"4d5a5f23-878d-436d-8f2a-7344d42779dd\" (UID: \"4d5a5f23-878d-436d-8f2a-7344d42779dd\") " Oct 31 13:30:22.223388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:22.223364 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5a5f23-878d-436d-8f2a-7344d42779dd-maas-api-service-ca" pod "4d5a5f23-878d-436d-8f2a-7344d42779dd" (UID: "4d5a5f23-878d-436d-8f2a-7344d42779dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:22.223730 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:22.223713 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d5a5f23-878d-436d-8f2a-7344d42779dd-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:22.225134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:22.225109 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5a5f23-878d-436d-8f2a-7344d42779dd-kube-api-access-88drw" pod "4d5a5f23-878d-436d-8f2a-7344d42779dd" (UID: "4d5a5f23-878d-436d-8f2a-7344d42779dd"). InnerVolumeSpecName "kube-api-access-88drw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:22.324858 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:22.324819 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88drw\" (UniqueName: \"kubernetes.io/projected/4d5a5f23-878d-436d-8f2a-7344d42779dd-kube-api-access-88drw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:22.848882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:22.848855 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" Oct 31 13:30:22.849313 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:22.848848 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2" event={"ID":"4d5a5f23-878d-436d-8f2a-7344d42779dd","Type":"ContainerDied","Data":"487eb236c09265b206ef55174d0cd58efa87061e8644b253b769430fdfdf13a8"} Oct 31 13:30:22.849313 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:22.848990 5532 scope.go:122] "RemoveContainer" containerID="20cbbc9f3d3752009f9141c59e03d7a9cca47cfffc0b327e9d0afddc74a31d5f" Oct 31 13:30:22.907116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:22.905770 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2"] Oct 31 13:30:22.911996 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:22.911966 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c82140-29890890-9sws2"] Oct 31 13:30:23.212505 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.212484 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" Oct 31 13:30:23.215039 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.215017 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" Oct 31 13:30:23.219101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.219083 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" Oct 31 13:30:23.223076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.223057 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" Oct 31 13:30:23.264460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.264428 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9799050e-794b-4634-9838-503737ce1cd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9799050e-794b-4634-9838-503737ce1cd0-maas-api-xks-ca\") pod \"9799050e-794b-4634-9838-503737ce1cd0\" (UID: \"9799050e-794b-4634-9838-503737ce1cd0\") " Oct 31 13:30:23.264690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.264555 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-kube-api-access-78p5c\" (UniqueName: \"kubernetes.io/projected/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-kube-api-access-78p5c\") pod \"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2\" (UID: \"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2\") " Oct 31 13:30:23.264806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.264588 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9799050e-794b-4634-9838-503737ce1cd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9799050e-794b-4634-9838-503737ce1cd0-maas-api-openshift-ca\") pod \"9799050e-794b-4634-9838-503737ce1cd0\" (UID: \"9799050e-794b-4634-9838-503737ce1cd0\") " Oct 31 13:30:23.264939 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.264911 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9799050e-794b-4634-9838-503737ce1cd0-maas-api-openshift-ca" pod "9799050e-794b-4634-9838-503737ce1cd0" (UID: "9799050e-794b-4634-9838-503737ce1cd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:23.265076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.265058 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-kube-api-access-l55ff\" (UniqueName: \"kubernetes.io/projected/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-kube-api-access-l55ff\") pod \"2bcf7019-e8dc-4f70-a37c-3e2640022b8c\" (UID: \"2bcf7019-e8dc-4f70-a37c-3e2640022b8c\") " Oct 31 13:30:23.265386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.265364 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08f66256-f121-47a7-b1a9-f6cb089802c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f66256-f121-47a7-b1a9-f6cb089802c0-maas-api-service-ca\") pod \"08f66256-f121-47a7-b1a9-f6cb089802c0\" (UID: \"08f66256-f121-47a7-b1a9-f6cb089802c0\") " Oct 31 13:30:23.265605 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.265573 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-maas-api-service-ca\") pod \"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2\" (UID: \"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2\") " Oct 31 13:30:23.265783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.265765 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-maas-api-service-ca\") pod \"2bcf7019-e8dc-4f70-a37c-3e2640022b8c\" (UID: \"2bcf7019-e8dc-4f70-a37c-3e2640022b8c\") " Oct 31 13:30:23.265958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.265942 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9799050e-794b-4634-9838-503737ce1cd0-kube-api-access-2bcm2\" (UniqueName: \"kubernetes.io/projected/9799050e-794b-4634-9838-503737ce1cd0-kube-api-access-2bcm2\") pod \"9799050e-794b-4634-9838-503737ce1cd0\" (UID: \"9799050e-794b-4634-9838-503737ce1cd0\") " Oct 31 13:30:23.266174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.266155 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08f66256-f121-47a7-b1a9-f6cb089802c0-kube-api-access-jlqw5\" (UniqueName: \"kubernetes.io/projected/08f66256-f121-47a7-b1a9-f6cb089802c0-kube-api-access-jlqw5\") pod \"08f66256-f121-47a7-b1a9-f6cb089802c0\" (UID: \"08f66256-f121-47a7-b1a9-f6cb089802c0\") " Oct 31 13:30:23.268061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.266373 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-maas-api-service-ca" pod "2bcf7019-e8dc-4f70-a37c-3e2640022b8c" (UID: "2bcf7019-e8dc-4f70-a37c-3e2640022b8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:23.268061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.267096 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9799050e-794b-4634-9838-503737ce1cd0-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:23.268061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.267121 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:23.268061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.267827 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08f66256-f121-47a7-b1a9-f6cb089802c0-maas-api-service-ca" pod "08f66256-f121-47a7-b1a9-f6cb089802c0" (UID: "08f66256-f121-47a7-b1a9-f6cb089802c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:23.268061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.267979 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-maas-api-service-ca" pod "cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" (UID: "cd813f2d-c7d1-4ad5-937f-2f78d6daebc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:23.268527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.268477 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9799050e-794b-4634-9838-503737ce1cd0-maas-api-xks-ca" pod "9799050e-794b-4634-9838-503737ce1cd0" (UID: "9799050e-794b-4634-9838-503737ce1cd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:30:23.268527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.268483 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-kube-api-access-l55ff" pod "2bcf7019-e8dc-4f70-a37c-3e2640022b8c" (UID: "2bcf7019-e8dc-4f70-a37c-3e2640022b8c"). InnerVolumeSpecName "kube-api-access-l55ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:23.268680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.268517 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-kube-api-access-78p5c" pod "cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" (UID: "cd813f2d-c7d1-4ad5-937f-2f78d6daebc2"). InnerVolumeSpecName "kube-api-access-78p5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:23.268916 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.268888 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08f66256-f121-47a7-b1a9-f6cb089802c0-kube-api-access-jlqw5" pod "08f66256-f121-47a7-b1a9-f6cb089802c0" (UID: "08f66256-f121-47a7-b1a9-f6cb089802c0"). InnerVolumeSpecName "kube-api-access-jlqw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:23.269475 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.269449 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9799050e-794b-4634-9838-503737ce1cd0-kube-api-access-2bcm2" pod "9799050e-794b-4634-9838-503737ce1cd0" (UID: "9799050e-794b-4634-9838-503737ce1cd0"). InnerVolumeSpecName "kube-api-access-2bcm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:23.368462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.368428 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9799050e-794b-4634-9838-503737ce1cd0-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:23.368462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.368458 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78p5c\" (UniqueName: \"kubernetes.io/projected/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-kube-api-access-78p5c\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:23.368462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.368469 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l55ff\" (UniqueName: \"kubernetes.io/projected/2bcf7019-e8dc-4f70-a37c-3e2640022b8c-kube-api-access-l55ff\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:23.368737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.368478 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08f66256-f121-47a7-b1a9-f6cb089802c0-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:23.368737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.368488 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:23.368737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.368501 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bcm2\" (UniqueName: \"kubernetes.io/projected/9799050e-794b-4634-9838-503737ce1cd0-kube-api-access-2bcm2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:23.368737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.368514 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlqw5\" (UniqueName: \"kubernetes.io/projected/08f66256-f121-47a7-b1a9-f6cb089802c0-kube-api-access-jlqw5\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:23.872690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.862862 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" event={"ID":"cd813f2d-c7d1-4ad5-937f-2f78d6daebc2","Type":"ContainerDied","Data":"5338d5a3e0eb4a5594abcc6b8d201c65a077be57238975ec8e7bf32cfbd1fc5c"} Oct 31 13:30:23.872690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.862934 5532 scope.go:122] "RemoveContainer" containerID="b3190e2445eb5570c569e49bb2f20409cc1627a440523ec192a3d3b77511869e" Oct 31 13:30:23.872690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.862871 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw" Oct 31 13:30:23.875650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.875622 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" Oct 31 13:30:23.879024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.878751 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" Oct 31 13:30:23.885719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.885683 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26" event={"ID":"2bcf7019-e8dc-4f70-a37c-3e2640022b8c","Type":"ContainerDied","Data":"85bf781cf89dfd910d5f77e107b22408f65ba963f6c87aaa6a635258a92a5c3b"} Oct 31 13:30:23.885880 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.885761 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q" event={"ID":"08f66256-f121-47a7-b1a9-f6cb089802c0","Type":"ContainerDied","Data":"52d731b47a81db013bab7c82ee6d1cbccd9b1a61928227b9d34e374b5e76a765"} Oct 31 13:30:23.886841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.886814 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" Oct 31 13:30:23.886959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.886849 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8" event={"ID":"9799050e-794b-4634-9838-503737ce1cd0","Type":"ContainerDied","Data":"c7db0532506b33c6c3e47417df1b740ab95016cded0f28a69b4b4dcf76db6431"} Oct 31 13:30:23.886959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.886916 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7db0532506b33c6c3e47417df1b740ab95016cded0f28a69b4b4dcf76db6431" Oct 31 13:30:23.900212 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.900163 5532 scope.go:122] "RemoveContainer" containerID="d8bc8df7cc1ec016365f934cded8af026188e4d13c1b2f38f619f05c74d30586" Oct 31 13:30:23.911272 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.911247 5532 scope.go:122] "RemoveContainer" containerID="ba2926805b3c95f7992e3cb2a9c5044e30d1f5f09ddf5c9e2f762a2efff85686" Oct 31 13:30:23.921817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.921780 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw"] Oct 31 13:30:23.939684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.939649 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-d85d82-29890890-6n6bw"] Oct 31 13:30:23.967227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.967193 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q"] Oct 31 13:30:23.972387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:23.972354 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-7b3ddd-29890890-g8p8q"] Oct 31 13:30:24.012542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:24.012508 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08f66256-f121-47a7-b1a9-f6cb089802c0" path="/var/lib/kubelet/pods/08f66256-f121-47a7-b1a9-f6cb089802c0/volumes" Oct 31 13:30:24.017205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:24.015007 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5a5f23-878d-436d-8f2a-7344d42779dd" path="/var/lib/kubelet/pods/4d5a5f23-878d-436d-8f2a-7344d42779dd/volumes" Oct 31 13:30:24.017205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:24.015881 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" path="/var/lib/kubelet/pods/cd813f2d-c7d1-4ad5-937f-2f78d6daebc2/volumes" Oct 31 13:30:24.017205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:24.016364 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26"] Oct 31 13:30:24.027587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:24.027562 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890890-lnr26"] Oct 31 13:30:25.552138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552102 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7"] Oct 31 13:30:25.552781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552766 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" containerName="cleanup" Oct 31 13:30:25.552781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552782 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" containerName="cleanup" Oct 31 13:30:25.552868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552790 5532 cpu_manager.go:414] "containerMap: removing container" podUID="08f66256-f121-47a7-b1a9-f6cb089802c0" containerName="cleanup" Oct 31 13:30:25.552868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552795 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f66256-f121-47a7-b1a9-f6cb089802c0" containerName="cleanup" Oct 31 13:30:25.552868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552806 5532 cpu_manager.go:414] "containerMap: removing container" podUID="08f66256-f121-47a7-b1a9-f6cb089802c0" containerName="cleanup" Oct 31 13:30:25.552868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552811 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f66256-f121-47a7-b1a9-f6cb089802c0" containerName="cleanup" Oct 31 13:30:25.552868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552826 5532 cpu_manager.go:414] "containerMap: removing container" podUID="08f66256-f121-47a7-b1a9-f6cb089802c0" containerName="cleanup" Oct 31 13:30:25.552868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552831 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08f66256-f121-47a7-b1a9-f6cb089802c0" containerName="cleanup" Oct 31 13:30:25.552868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552838 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4d5a5f23-878d-436d-8f2a-7344d42779dd" containerName="cleanup" Oct 31 13:30:25.552868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552843 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5a5f23-878d-436d-8f2a-7344d42779dd" containerName="cleanup" Oct 31 13:30:25.552868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552850 5532 cpu_manager.go:414] "containerMap: removing container" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" containerName="cleanup" Oct 31 13:30:25.552868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552855 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552879 5532 cpu_manager.go:414] "containerMap: removing container" podUID="9799050e-794b-4634-9838-503737ce1cd0" containerName="revoke-keys" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552883 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9799050e-794b-4634-9838-503737ce1cd0" containerName="revoke-keys" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552901 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4d5a5f23-878d-436d-8f2a-7344d42779dd" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552905 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5a5f23-878d-436d-8f2a-7344d42779dd" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552915 5532 cpu_manager.go:414] "containerMap: removing container" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552919 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552928 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.552932 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.553034 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.553041 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5a5f23-878d-436d-8f2a-7344d42779dd" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.553049 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5a5f23-878d-436d-8f2a-7344d42779dd" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.553057 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.553063 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.553069 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f66256-f121-47a7-b1a9-f6cb089802c0" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.553074 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f66256-f121-47a7-b1a9-f6cb089802c0" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.553083 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.553093 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" containerName="cleanup" Oct 31 13:30:25.553142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.553099 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="9799050e-794b-4634-9838-503737ce1cd0" containerName="revoke-keys" Oct 31 13:30:25.558452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.558432 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" Oct 31 13:30:25.563767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.563737 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:30:25.565015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.564993 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:30:25.568302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.567957 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7"] Oct 31 13:30:25.610757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.610729 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sfbv\" (UniqueName: \"kubernetes.io/projected/f961cdee-55f0-4b63-b43e-0394108e51d1-kube-api-access-6sfbv\") pod \"maas-api-revoke-sub-73999081-68vp7\" (UID: \"f961cdee-55f0-4b63-b43e-0394108e51d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" Oct 31 13:30:25.610923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.610765 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73999081-68vp7\" (UID: \"f961cdee-55f0-4b63-b43e-0394108e51d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" Oct 31 13:30:25.610923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.610883 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73999081-68vp7\" (UID: \"f961cdee-55f0-4b63-b43e-0394108e51d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" Oct 31 13:30:25.712949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.712915 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sfbv\" (UniqueName: \"kubernetes.io/projected/f961cdee-55f0-4b63-b43e-0394108e51d1-kube-api-access-6sfbv\") pod \"maas-api-revoke-sub-73999081-68vp7\" (UID: \"f961cdee-55f0-4b63-b43e-0394108e51d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" Oct 31 13:30:25.713144 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.712960 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73999081-68vp7\" (UID: \"f961cdee-55f0-4b63-b43e-0394108e51d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" Oct 31 13:30:25.713144 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.713019 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73999081-68vp7\" (UID: \"f961cdee-55f0-4b63-b43e-0394108e51d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" Oct 31 13:30:25.713920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.713885 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73999081-68vp7\" (UID: \"f961cdee-55f0-4b63-b43e-0394108e51d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" Oct 31 13:30:25.715530 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.715500 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73999081-68vp7\" (UID: \"f961cdee-55f0-4b63-b43e-0394108e51d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" Oct 31 13:30:25.726483 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.726458 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sfbv\" (UniqueName: \"kubernetes.io/projected/f961cdee-55f0-4b63-b43e-0394108e51d1-kube-api-access-6sfbv\") pod \"maas-api-revoke-sub-73999081-68vp7\" (UID: \"f961cdee-55f0-4b63-b43e-0394108e51d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" Oct 31 13:30:25.876920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:25.876835 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" Oct 31 13:30:26.007535 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.007503 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" path="/var/lib/kubelet/pods/2bcf7019-e8dc-4f70-a37c-3e2640022b8c/volumes" Oct 31 13:30:26.167241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.167204 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7"] Oct 31 13:30:26.173335 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:30:26.173300 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf961cdee_55f0_4b63_b43e_0394108e51d1.slice/crio-653193fd27afd177940b0c57726ec7e86ef5c4da70ae610339269cb65ecaeacb WatchSource:0}: Error finding container 653193fd27afd177940b0c57726ec7e86ef5c4da70ae610339269cb65ecaeacb: Status 404 returned error can't find the container with id 653193fd27afd177940b0c57726ec7e86ef5c4da70ae610339269cb65ecaeacb Oct 31 13:30:26.766747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.766674 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278"] Oct 31 13:30:26.767364 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.767350 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4d5a5f23-878d-436d-8f2a-7344d42779dd" containerName="cleanup" Oct 31 13:30:26.767403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.767365 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5a5f23-878d-436d-8f2a-7344d42779dd" containerName="cleanup" Oct 31 13:30:26.767446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.767410 5532 cpu_manager.go:414] "containerMap: removing container" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" containerName="cleanup" Oct 31 13:30:26.767446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.767416 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd813f2d-c7d1-4ad5-937f-2f78d6daebc2" containerName="cleanup" Oct 31 13:30:26.767446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.767429 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" containerName="cleanup" Oct 31 13:30:26.767446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.767435 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" containerName="cleanup" Oct 31 13:30:26.767570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.767531 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5a5f23-878d-436d-8f2a-7344d42779dd" containerName="cleanup" Oct 31 13:30:26.767570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.767542 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcf7019-e8dc-4f70-a37c-3e2640022b8c" containerName="cleanup" Oct 31 13:30:26.767570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.767550 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="08f66256-f121-47a7-b1a9-f6cb089802c0" containerName="cleanup" Oct 31 13:30:26.771422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.771404 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" Oct 31 13:30:26.798898 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.798870 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278"] Oct 31 13:30:26.831719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.831691 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtfff\" (UniqueName: \"kubernetes.io/projected/45325bfe-233f-47ca-8457-8f6b197ec554-kube-api-access-qtfff\") pod \"maas-api-revoke-sub-ceeaf5ef-72278\" (UID: \"45325bfe-233f-47ca-8457-8f6b197ec554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" Oct 31 13:30:26.831958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.831939 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceeaf5ef-72278\" (UID: \"45325bfe-233f-47ca-8457-8f6b197ec554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" Oct 31 13:30:26.832035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.831990 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceeaf5ef-72278\" (UID: \"45325bfe-233f-47ca-8457-8f6b197ec554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" Oct 31 13:30:26.920695 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.920654 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" event={"ID":"f961cdee-55f0-4b63-b43e-0394108e51d1","Type":"ContainerStarted","Data":"0c186ca8d9c1ddb46fd3113a217301f76d290fbecad95da44f05a5dba548bf84"} Oct 31 13:30:26.920848 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.920717 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" event={"ID":"f961cdee-55f0-4b63-b43e-0394108e51d1","Type":"ContainerStarted","Data":"653193fd27afd177940b0c57726ec7e86ef5c4da70ae610339269cb65ecaeacb"} Oct 31 13:30:26.933745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.933716 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceeaf5ef-72278\" (UID: \"45325bfe-233f-47ca-8457-8f6b197ec554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" Oct 31 13:30:26.933838 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.933777 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceeaf5ef-72278\" (UID: \"45325bfe-233f-47ca-8457-8f6b197ec554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" Oct 31 13:30:26.933915 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.933900 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtfff\" (UniqueName: \"kubernetes.io/projected/45325bfe-233f-47ca-8457-8f6b197ec554-kube-api-access-qtfff\") pod \"maas-api-revoke-sub-ceeaf5ef-72278\" (UID: \"45325bfe-233f-47ca-8457-8f6b197ec554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" Oct 31 13:30:26.934523 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.934493 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ceeaf5ef-72278\" (UID: \"45325bfe-233f-47ca-8457-8f6b197ec554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" Oct 31 13:30:26.936216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.936189 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ceeaf5ef-72278\" (UID: \"45325bfe-233f-47ca-8457-8f6b197ec554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" Oct 31 13:30:26.947389 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.947365 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtfff\" (UniqueName: \"kubernetes.io/projected/45325bfe-233f-47ca-8457-8f6b197ec554-kube-api-access-qtfff\") pod \"maas-api-revoke-sub-ceeaf5ef-72278\" (UID: \"45325bfe-233f-47ca-8457-8f6b197ec554\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" Oct 31 13:30:26.951415 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:26.951364 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" podStartSLOduration=1.951346854 podStartE2EDuration="1.951346854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:26.950841974 +0000 UTC m=+1595.096585804" watchObservedRunningTime="2026-10-31 13:30:26.951346854 +0000 UTC m=+1595.097090683" Oct 31 13:30:27.089744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.089651 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" Oct 31 13:30:27.376921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.376841 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278"] Oct 31 13:30:27.386110 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:30:27.386077 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45325bfe_233f_47ca_8457_8f6b197ec554.slice/crio-13603b53690e65196b4892db0a46713214ca2390d8b7c7fb0b06e1f079baffb0 WatchSource:0}: Error finding container 13603b53690e65196b4892db0a46713214ca2390d8b7c7fb0b06e1f079baffb0: Status 404 returned error can't find the container with id 13603b53690e65196b4892db0a46713214ca2390d8b7c7fb0b06e1f079baffb0 Oct 31 13:30:27.743963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.743922 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk"] Oct 31 13:30:27.752288 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.752258 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" Oct 31 13:30:27.763148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.763118 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk"] Oct 31 13:30:27.854370 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.854334 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgb2p\" (UniqueName: \"kubernetes.io/projected/d58c3e46-103b-408d-a0a0-b4fd24891dfc-kube-api-access-hgb2p\") pod \"maas-api-revoke-sub-17c30975-bnqmk\" (UID: \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" Oct 31 13:30:27.854790 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.854487 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17c30975-bnqmk\" (UID: \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" Oct 31 13:30:27.854832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.854795 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17c30975-bnqmk\" (UID: \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" Oct 31 13:30:27.932484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.932398 5532 generic.go:356] "Generic (PLEG): container finished" podID="f961cdee-55f0-4b63-b43e-0394108e51d1" containerID="0c186ca8d9c1ddb46fd3113a217301f76d290fbecad95da44f05a5dba548bf84" exitCode=0 Oct 31 13:30:27.932738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.932693 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" event={"ID":"f961cdee-55f0-4b63-b43e-0394108e51d1","Type":"ContainerDied","Data":"0c186ca8d9c1ddb46fd3113a217301f76d290fbecad95da44f05a5dba548bf84"} Oct 31 13:30:27.934833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.934803 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" event={"ID":"45325bfe-233f-47ca-8457-8f6b197ec554","Type":"ContainerStarted","Data":"96160a93b2d259704b759b7ad2667625f376515a92028e4ccf1c6bb2db2c4722"} Oct 31 13:30:27.934932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.934857 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" event={"ID":"45325bfe-233f-47ca-8457-8f6b197ec554","Type":"ContainerStarted","Data":"13603b53690e65196b4892db0a46713214ca2390d8b7c7fb0b06e1f079baffb0"} Oct 31 13:30:27.937143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.937121 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5"] Oct 31 13:30:27.942085 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.942066 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" Oct 31 13:30:27.952851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.952719 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5"] Oct 31 13:30:27.957644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.957576 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17c30975-bnqmk\" (UID: \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" Oct 31 13:30:27.957768 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.957643 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgb2p\" (UniqueName: \"kubernetes.io/projected/d58c3e46-103b-408d-a0a0-b4fd24891dfc-kube-api-access-hgb2p\") pod \"maas-api-revoke-sub-17c30975-bnqmk\" (UID: \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" Oct 31 13:30:27.957768 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.957745 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17c30975-bnqmk\" (UID: \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" Oct 31 13:30:27.958376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.958352 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17c30975-bnqmk\" (UID: \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" Oct 31 13:30:27.963274 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.963246 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17c30975-bnqmk\" (UID: \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" Oct 31 13:30:27.978532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.978500 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgb2p\" (UniqueName: \"kubernetes.io/projected/d58c3e46-103b-408d-a0a0-b4fd24891dfc-kube-api-access-hgb2p\") pod \"maas-api-revoke-sub-17c30975-bnqmk\" (UID: \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" Oct 31 13:30:27.986874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:27.986827 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" podStartSLOduration=1.9868163989999998 podStartE2EDuration="1.986816399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:27.980993618 +0000 UTC m=+1596.126737448" watchObservedRunningTime="2026-10-31 13:30:27.986816399 +0000 UTC m=+1596.132560227" Oct 31 13:30:28.059421 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.059387 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2680b5e-kjqs5\" (UID: \"874d4baf-d227-49f6-9b75-7e18e1a729f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" Oct 31 13:30:28.059975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.059721 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2680b5e-kjqs5\" (UID: \"874d4baf-d227-49f6-9b75-7e18e1a729f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" Oct 31 13:30:28.060132 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.060107 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b24n\" (UniqueName: \"kubernetes.io/projected/874d4baf-d227-49f6-9b75-7e18e1a729f1-kube-api-access-7b24n\") pod \"maas-api-revoke-sub-f2680b5e-kjqs5\" (UID: \"874d4baf-d227-49f6-9b75-7e18e1a729f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" Oct 31 13:30:28.078418 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.078393 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" Oct 31 13:30:28.163646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.162997 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2680b5e-kjqs5\" (UID: \"874d4baf-d227-49f6-9b75-7e18e1a729f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" Oct 31 13:30:28.163646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.163213 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b24n\" (UniqueName: \"kubernetes.io/projected/874d4baf-d227-49f6-9b75-7e18e1a729f1-kube-api-access-7b24n\") pod \"maas-api-revoke-sub-f2680b5e-kjqs5\" (UID: \"874d4baf-d227-49f6-9b75-7e18e1a729f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" Oct 31 13:30:28.163646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.163367 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2680b5e-kjqs5\" (UID: \"874d4baf-d227-49f6-9b75-7e18e1a729f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" Oct 31 13:30:28.163908 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.163842 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2680b5e-kjqs5\" (UID: \"874d4baf-d227-49f6-9b75-7e18e1a729f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" Oct 31 13:30:28.165684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.165659 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2680b5e-kjqs5\" (UID: \"874d4baf-d227-49f6-9b75-7e18e1a729f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" Oct 31 13:30:28.176671 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.176647 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b24n\" (UniqueName: \"kubernetes.io/projected/874d4baf-d227-49f6-9b75-7e18e1a729f1-kube-api-access-7b24n\") pod \"maas-api-revoke-sub-f2680b5e-kjqs5\" (UID: \"874d4baf-d227-49f6-9b75-7e18e1a729f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" Oct 31 13:30:28.265089 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.265059 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" Oct 31 13:30:28.388749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.377163 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk"] Oct 31 13:30:28.588253 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:30:28.588200 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874d4baf_d227_49f6_9b75_7e18e1a729f1.slice/crio-cd1d3f3cb649d43e6427474c56b354a23b3866c0cfffd730ed61d6434aff359e WatchSource:0}: Error finding container cd1d3f3cb649d43e6427474c56b354a23b3866c0cfffd730ed61d6434aff359e: Status 404 returned error can't find the container with id cd1d3f3cb649d43e6427474c56b354a23b3866c0cfffd730ed61d6434aff359e Oct 31 13:30:28.589807 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.589535 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5"] Oct 31 13:30:28.944976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.944943 5532 generic.go:356] "Generic (PLEG): container finished" podID="45325bfe-233f-47ca-8457-8f6b197ec554" containerID="96160a93b2d259704b759b7ad2667625f376515a92028e4ccf1c6bb2db2c4722" exitCode=0 Oct 31 13:30:28.945367 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.945136 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" event={"ID":"45325bfe-233f-47ca-8457-8f6b197ec554","Type":"ContainerDied","Data":"96160a93b2d259704b759b7ad2667625f376515a92028e4ccf1c6bb2db2c4722"} Oct 31 13:30:28.947042 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.947009 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" event={"ID":"d58c3e46-103b-408d-a0a0-b4fd24891dfc","Type":"ContainerStarted","Data":"85c82577da11c8e35cce26cb614b6fbe4d61378d8b950ad4a7a3914f9dc67287"} Oct 31 13:30:28.947182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.947069 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" event={"ID":"d58c3e46-103b-408d-a0a0-b4fd24891dfc","Type":"ContainerStarted","Data":"b433614acb52b5e76dbd8d46aca845900d363d494a1e7da48feb5c8f38b3db2e"} Oct 31 13:30:28.948729 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.948702 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" event={"ID":"874d4baf-d227-49f6-9b75-7e18e1a729f1","Type":"ContainerStarted","Data":"cd1d3f3cb649d43e6427474c56b354a23b3866c0cfffd730ed61d6434aff359e"} Oct 31 13:30:28.989450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:28.989396 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" podStartSLOduration=1.9893803220000001 podStartE2EDuration="1.989380322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:28.989183043 +0000 UTC m=+1597.134926870" watchObservedRunningTime="2026-10-31 13:30:28.989380322 +0000 UTC m=+1597.135124150" Oct 31 13:30:29.137779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.137744 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4"] Oct 31 13:30:29.143924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.143897 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.145631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.145561 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" Oct 31 13:30:29.156513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.156482 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4"] Oct 31 13:30:29.156859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.156838 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c7b990ba-openshift-default-dockercfg-r2b9f\"" Oct 31 13:30:29.295138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.295049 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f961cdee-55f0-4b63-b43e-0394108e51d1-kube-api-access-6sfbv\" (UniqueName: \"kubernetes.io/projected/f961cdee-55f0-4b63-b43e-0394108e51d1-kube-api-access-6sfbv\") pod \"f961cdee-55f0-4b63-b43e-0394108e51d1\" (UID: \"f961cdee-55f0-4b63-b43e-0394108e51d1\") " Oct 31 13:30:29.295138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.295110 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-xks-ca\") pod \"f961cdee-55f0-4b63-b43e-0394108e51d1\" (UID: \"f961cdee-55f0-4b63-b43e-0394108e51d1\") " Oct 31 13:30:29.295370 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.295219 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-openshift-ca\") pod \"f961cdee-55f0-4b63-b43e-0394108e51d1\" (UID: \"f961cdee-55f0-4b63-b43e-0394108e51d1\") " Oct 31 13:30:29.295575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.295543 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-openshift-ca" pod "f961cdee-55f0-4b63-b43e-0394108e51d1" (UID: "f961cdee-55f0-4b63-b43e-0394108e51d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:29.295826 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.295804 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t876h\" (UniqueName: \"kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-kube-api-access-t876h\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.295913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.295846 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255e3805-e523-4620-b06a-f4d8fa91538f-istiod-ca-cert\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.295998 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.295984 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255e3805-e523-4620-b06a-f4d8fa91538f-istio-podinfo\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.296061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.296052 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-envoy\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.296125 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.296073 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-certs\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.296187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.296142 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-istio-token\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.296247 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.296196 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-data\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.296247 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.296219 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-socket\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.296386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.296360 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-credential-socket\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.296556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.296537 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:29.297416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.297389 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-xks-ca" pod "f961cdee-55f0-4b63-b43e-0394108e51d1" (UID: "f961cdee-55f0-4b63-b43e-0394108e51d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:30:29.297505 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.297440 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f961cdee-55f0-4b63-b43e-0394108e51d1-kube-api-access-6sfbv" pod "f961cdee-55f0-4b63-b43e-0394108e51d1" (UID: "f961cdee-55f0-4b63-b43e-0394108e51d1"). InnerVolumeSpecName "kube-api-access-6sfbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:29.407626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.406480 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-data\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.407626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.406545 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-socket\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.407626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.406634 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-credential-socket\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.407626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.406848 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t876h\" (UniqueName: \"kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-kube-api-access-t876h\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.407626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.406893 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255e3805-e523-4620-b06a-f4d8fa91538f-istiod-ca-cert\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.407626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.406986 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255e3805-e523-4620-b06a-f4d8fa91538f-istio-podinfo\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.407626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.407044 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-envoy\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.407626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.407074 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-certs\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.407626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.407126 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-istio-token\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.407626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.407225 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sfbv\" (UniqueName: \"kubernetes.io/projected/f961cdee-55f0-4b63-b43e-0394108e51d1-kube-api-access-6sfbv\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:29.407626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.407241 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f961cdee-55f0-4b63-b43e-0394108e51d1-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:29.408354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.407882 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-data\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.408354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.408148 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-socket\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.408457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.408395 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-credential-socket\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.410684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.410373 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255e3805-e523-4620-b06a-f4d8fa91538f-istiod-ca-cert\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.410684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.410657 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-certs\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.415797 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.415464 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255e3805-e523-4620-b06a-f4d8fa91538f-istio-podinfo\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.444636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.441399 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-envoy\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.470150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.469320 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-istio-token\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.487642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.487291 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t876h\" (UniqueName: \"kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-kube-api-access-t876h\") pod \"e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.768509 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.768477 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:29.958208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.958168 5532 generic.go:356] "Generic (PLEG): container finished" podID="874d4baf-d227-49f6-9b75-7e18e1a729f1" containerID="46d8892cd4b13f4e625f5fcefc4675e0483784fba9b8b3c895965300e97309c3" exitCode=0 Oct 31 13:30:29.958694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.958254 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" event={"ID":"874d4baf-d227-49f6-9b75-7e18e1a729f1","Type":"ContainerDied","Data":"46d8892cd4b13f4e625f5fcefc4675e0483784fba9b8b3c895965300e97309c3"} Oct 31 13:30:29.959791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.959770 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" Oct 31 13:30:29.959791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.959777 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7" event={"ID":"f961cdee-55f0-4b63-b43e-0394108e51d1","Type":"ContainerDied","Data":"653193fd27afd177940b0c57726ec7e86ef5c4da70ae610339269cb65ecaeacb"} Oct 31 13:30:29.959960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.959819 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="653193fd27afd177940b0c57726ec7e86ef5c4da70ae610339269cb65ecaeacb" Oct 31 13:30:29.961555 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.961530 5532 generic.go:356] "Generic (PLEG): container finished" podID="d58c3e46-103b-408d-a0a0-b4fd24891dfc" containerID="85c82577da11c8e35cce26cb614b6fbe4d61378d8b950ad4a7a3914f9dc67287" exitCode=0 Oct 31 13:30:29.961717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:29.961614 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" event={"ID":"d58c3e46-103b-408d-a0a0-b4fd24891dfc","Type":"ContainerDied","Data":"85c82577da11c8e35cce26cb614b6fbe4d61378d8b950ad4a7a3914f9dc67287"} Oct 31 13:30:30.072360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.072322 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4"] Oct 31 13:30:30.073788 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:30:30.073727 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod255e3805_e523_4620_b06a_f4d8fa91538f.slice/crio-1a9d5cb30f9a829d9175eb9fc011d0270b1a8be9404199e01e712c4162958064 WatchSource:0}: Error finding container 1a9d5cb30f9a829d9175eb9fc011d0270b1a8be9404199e01e712c4162958064: Status 404 returned error can't find the container with id 1a9d5cb30f9a829d9175eb9fc011d0270b1a8be9404199e01e712c4162958064 Oct 31 13:30:30.076451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.076414 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:30:30.076558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.076502 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:30:30.076558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.076550 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:30:30.145805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.145781 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" Oct 31 13:30:30.325571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.325478 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45325bfe-233f-47ca-8457-8f6b197ec554-kube-api-access-qtfff\" (UniqueName: \"kubernetes.io/projected/45325bfe-233f-47ca-8457-8f6b197ec554-kube-api-access-qtfff\") pod \"45325bfe-233f-47ca-8457-8f6b197ec554\" (UID: \"45325bfe-233f-47ca-8457-8f6b197ec554\") " Oct 31 13:30:30.325571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.325532 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-xks-ca\") pod \"45325bfe-233f-47ca-8457-8f6b197ec554\" (UID: \"45325bfe-233f-47ca-8457-8f6b197ec554\") " Oct 31 13:30:30.325830 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.325589 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-openshift-ca\") pod \"45325bfe-233f-47ca-8457-8f6b197ec554\" (UID: \"45325bfe-233f-47ca-8457-8f6b197ec554\") " Oct 31 13:30:30.325979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.325948 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-openshift-ca" pod "45325bfe-233f-47ca-8457-8f6b197ec554" (UID: "45325bfe-233f-47ca-8457-8f6b197ec554"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:30.326630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.326574 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:30.343231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.328246 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-xks-ca" pod "45325bfe-233f-47ca-8457-8f6b197ec554" (UID: "45325bfe-233f-47ca-8457-8f6b197ec554"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:30:30.343231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.328275 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45325bfe-233f-47ca-8457-8f6b197ec554-kube-api-access-qtfff" pod "45325bfe-233f-47ca-8457-8f6b197ec554" (UID: "45325bfe-233f-47ca-8457-8f6b197ec554"). InnerVolumeSpecName "kube-api-access-qtfff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:30.428663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.428629 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtfff\" (UniqueName: \"kubernetes.io/projected/45325bfe-233f-47ca-8457-8f6b197ec554-kube-api-access-qtfff\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:30.428663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.428663 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45325bfe-233f-47ca-8457-8f6b197ec554-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:30.970876 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.970843 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" Oct 31 13:30:30.970876 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.970871 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278" event={"ID":"45325bfe-233f-47ca-8457-8f6b197ec554","Type":"ContainerDied","Data":"13603b53690e65196b4892db0a46713214ca2390d8b7c7fb0b06e1f079baffb0"} Oct 31 13:30:30.971321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.970916 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="13603b53690e65196b4892db0a46713214ca2390d8b7c7fb0b06e1f079baffb0" Oct 31 13:30:30.972847 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.972819 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" event={"ID":"255e3805-e523-4620-b06a-f4d8fa91538f","Type":"ContainerStarted","Data":"b5c85baeaf8e3bc4d53903705543dfe2296d252cd396456529c5676604641255"} Oct 31 13:30:30.972988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:30.972864 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" event={"ID":"255e3805-e523-4620-b06a-f4d8fa91538f","Type":"ContainerStarted","Data":"1a9d5cb30f9a829d9175eb9fc011d0270b1a8be9404199e01e712c4162958064"} Oct 31 13:30:31.012960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.012893 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" podStartSLOduration=2.012872859 podStartE2EDuration="2.012872859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:31.00402809 +0000 UTC m=+1599.149771933" watchObservedRunningTime="2026-10-31 13:30:31.012872859 +0000 UTC m=+1599.158616687" Oct 31 13:30:31.345455 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.345123 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" Oct 31 13:30:31.369615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.362835 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" Oct 31 13:30:31.446731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.446687 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-openshift-ca\") pod \"874d4baf-d227-49f6-9b75-7e18e1a729f1\" (UID: \"874d4baf-d227-49f6-9b75-7e18e1a729f1\") " Oct 31 13:30:31.446954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.446745 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-openshift-ca\") pod \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\" (UID: \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\") " Oct 31 13:30:31.446954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.446773 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/874d4baf-d227-49f6-9b75-7e18e1a729f1-kube-api-access-7b24n\" (UniqueName: \"kubernetes.io/projected/874d4baf-d227-49f6-9b75-7e18e1a729f1-kube-api-access-7b24n\") pod \"874d4baf-d227-49f6-9b75-7e18e1a729f1\" (UID: \"874d4baf-d227-49f6-9b75-7e18e1a729f1\") " Oct 31 13:30:31.446954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.446804 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d58c3e46-103b-408d-a0a0-b4fd24891dfc-kube-api-access-hgb2p\" (UniqueName: \"kubernetes.io/projected/d58c3e46-103b-408d-a0a0-b4fd24891dfc-kube-api-access-hgb2p\") pod \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\" (UID: \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\") " Oct 31 13:30:31.446954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.446885 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-xks-ca\") pod \"874d4baf-d227-49f6-9b75-7e18e1a729f1\" (UID: \"874d4baf-d227-49f6-9b75-7e18e1a729f1\") " Oct 31 13:30:31.447136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.447032 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-xks-ca\") pod \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\" (UID: \"d58c3e46-103b-408d-a0a0-b4fd24891dfc\") " Oct 31 13:30:31.450147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.448223 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-openshift-ca" pod "d58c3e46-103b-408d-a0a0-b4fd24891dfc" (UID: "d58c3e46-103b-408d-a0a0-b4fd24891dfc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:31.450147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.448907 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-openshift-ca" pod "874d4baf-d227-49f6-9b75-7e18e1a729f1" (UID: "874d4baf-d227-49f6-9b75-7e18e1a729f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:31.450147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.450002 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d58c3e46-103b-408d-a0a0-b4fd24891dfc-kube-api-access-hgb2p" pod "d58c3e46-103b-408d-a0a0-b4fd24891dfc" (UID: "d58c3e46-103b-408d-a0a0-b4fd24891dfc"). InnerVolumeSpecName "kube-api-access-hgb2p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:31.450147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.450084 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/874d4baf-d227-49f6-9b75-7e18e1a729f1-kube-api-access-7b24n" pod "874d4baf-d227-49f6-9b75-7e18e1a729f1" (UID: "874d4baf-d227-49f6-9b75-7e18e1a729f1"). InnerVolumeSpecName "kube-api-access-7b24n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:31.450147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.450141 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-xks-ca" pod "874d4baf-d227-49f6-9b75-7e18e1a729f1" (UID: "874d4baf-d227-49f6-9b75-7e18e1a729f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:30:31.450382 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.450196 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-xks-ca" pod "d58c3e46-103b-408d-a0a0-b4fd24891dfc" (UID: "d58c3e46-103b-408d-a0a0-b4fd24891dfc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:30:31.549008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.548925 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:31.549008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.548956 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:31.549008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.548965 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d58c3e46-103b-408d-a0a0-b4fd24891dfc-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:31.549008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.548974 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b24n\" (UniqueName: \"kubernetes.io/projected/874d4baf-d227-49f6-9b75-7e18e1a729f1-kube-api-access-7b24n\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:31.549008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.548983 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgb2p\" (UniqueName: \"kubernetes.io/projected/d58c3e46-103b-408d-a0a0-b4fd24891dfc-kube-api-access-hgb2p\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:31.549008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.548993 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/874d4baf-d227-49f6-9b75-7e18e1a729f1-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:31.762618 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:31.762565 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:30:31.768855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.768832 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:31.774833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.774809 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:31.981776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.981742 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" Oct 31 13:30:31.982267 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.981776 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5" event={"ID":"874d4baf-d227-49f6-9b75-7e18e1a729f1","Type":"ContainerDied","Data":"cd1d3f3cb649d43e6427474c56b354a23b3866c0cfffd730ed61d6434aff359e"} Oct 31 13:30:31.982267 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.981836 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd1d3f3cb649d43e6427474c56b354a23b3866c0cfffd730ed61d6434aff359e" Oct 31 13:30:31.983406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.983378 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" event={"ID":"d58c3e46-103b-408d-a0a0-b4fd24891dfc","Type":"ContainerDied","Data":"b433614acb52b5e76dbd8d46aca845900d363d494a1e7da48feb5c8f38b3db2e"} Oct 31 13:30:31.983406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.983404 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk" Oct 31 13:30:31.983572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.983423 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b433614acb52b5e76dbd8d46aca845900d363d494a1e7da48feb5c8f38b3db2e" Oct 31 13:30:31.983772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.983761 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:31.984636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:31.984612 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:30:37.248770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:37.248730 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:30:38.006867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.006834 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r"] Oct 31 13:30:38.007663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.007641 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f961cdee-55f0-4b63-b43e-0394108e51d1" containerName="revoke-keys" Oct 31 13:30:38.007663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.007660 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f961cdee-55f0-4b63-b43e-0394108e51d1" containerName="revoke-keys" Oct 31 13:30:38.007862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.007672 5532 cpu_manager.go:414] "containerMap: removing container" podUID="874d4baf-d227-49f6-9b75-7e18e1a729f1" containerName="revoke-keys" Oct 31 13:30:38.007862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.007678 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="874d4baf-d227-49f6-9b75-7e18e1a729f1" containerName="revoke-keys" Oct 31 13:30:38.007862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.007706 5532 cpu_manager.go:414] "containerMap: removing container" podUID="45325bfe-233f-47ca-8457-8f6b197ec554" containerName="revoke-keys" Oct 31 13:30:38.007862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.007714 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45325bfe-233f-47ca-8457-8f6b197ec554" containerName="revoke-keys" Oct 31 13:30:38.007862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.007728 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d58c3e46-103b-408d-a0a0-b4fd24891dfc" containerName="revoke-keys" Oct 31 13:30:38.007862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.007733 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d58c3e46-103b-408d-a0a0-b4fd24891dfc" containerName="revoke-keys" Oct 31 13:30:38.008111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.007903 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f961cdee-55f0-4b63-b43e-0394108e51d1" containerName="revoke-keys" Oct 31 13:30:38.008111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.007913 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="874d4baf-d227-49f6-9b75-7e18e1a729f1" containerName="revoke-keys" Oct 31 13:30:38.008111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.007921 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="45325bfe-233f-47ca-8457-8f6b197ec554" containerName="revoke-keys" Oct 31 13:30:38.008111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.007929 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d58c3e46-103b-408d-a0a0-b4fd24891dfc" containerName="revoke-keys" Oct 31 13:30:38.015864 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.015305 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.020015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.019988 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r"] Oct 31 13:30:38.027217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.027170 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-fc1bbe2b-gw-openshift-default-dockercfg-pbx8w\"" Oct 31 13:30:38.067549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.065037 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-istio-token\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.067549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.065094 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-credential-socket\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.067549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.065257 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-data\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.067549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.065411 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-envoy\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.067549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.065639 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204317d2-4400-4923-adac-e410dced28b3-istio-podinfo\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.067549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.065728 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-certs\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.067549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.065798 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfc5n\" (UniqueName: \"kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-kube-api-access-nfc5n\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.067549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.066025 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204317d2-4400-4923-adac-e410dced28b3-istiod-ca-cert\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.067549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.066140 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-socket\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.109862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.109831 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr"] Oct 31 13:30:38.114535 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.114514 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" Oct 31 13:30:38.131918 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.131892 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr"] Oct 31 13:30:38.134812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.134747 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:30:38.134925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.134907 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:30:38.167938 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.167879 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfc5n\" (UniqueName: \"kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-kube-api-access-nfc5n\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.168187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.167974 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1447791b-lh9jr\" (UID: \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" Oct 31 13:30:38.168187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.168005 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stz24\" (UniqueName: \"kubernetes.io/projected/cdde2ce8-b77f-48c8-ab0a-808aab30216c-kube-api-access-stz24\") pod \"maas-api-revoke-sub-1447791b-lh9jr\" (UID: \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" Oct 31 13:30:38.168187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.168074 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204317d2-4400-4923-adac-e410dced28b3-istiod-ca-cert\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.168187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.168169 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-socket\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.168476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.168238 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-istio-token\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.168476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.168281 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-credential-socket\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.168476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.168367 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-data\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.168476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.168395 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-envoy\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.168476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.168457 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1447791b-lh9jr\" (UID: \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" Oct 31 13:30:38.168476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.168493 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204317d2-4400-4923-adac-e410dced28b3-istio-podinfo\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.168983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.168573 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-certs\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.169140 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.169112 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-certs\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.170191 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.170157 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204317d2-4400-4923-adac-e410dced28b3-istiod-ca-cert\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.170437 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.170415 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-socket\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.171049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.171022 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-credential-socket\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.171376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.171332 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-data\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.175835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.175765 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204317d2-4400-4923-adac-e410dced28b3-istio-podinfo\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.223628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.202344 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfc5n\" (UniqueName: \"kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-kube-api-access-nfc5n\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.223628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.204039 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-envoy\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.224520 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.224449 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-istio-token\") pod \"e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.312862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.312732 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1447791b-lh9jr\" (UID: \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" Oct 31 13:30:38.332087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.311785 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1447791b-lh9jr\" (UID: \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" Oct 31 13:30:38.332087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.321611 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stz24\" (UniqueName: \"kubernetes.io/projected/cdde2ce8-b77f-48c8-ab0a-808aab30216c-kube-api-access-stz24\") pod \"maas-api-revoke-sub-1447791b-lh9jr\" (UID: \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" Oct 31 13:30:38.332087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.321981 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1447791b-lh9jr\" (UID: \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" Oct 31 13:30:38.336588 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.336113 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:38.352383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.352048 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1447791b-lh9jr\" (UID: \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" Oct 31 13:30:38.479619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.429389 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stz24\" (UniqueName: \"kubernetes.io/projected/cdde2ce8-b77f-48c8-ab0a-808aab30216c-kube-api-access-stz24\") pod \"maas-api-revoke-sub-1447791b-lh9jr\" (UID: \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" Oct 31 13:30:38.479619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:38.431307 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" Oct 31 13:30:39.086435 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:39.084099 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r"] Oct 31 13:30:39.106730 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:39.105850 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:30:39.106730 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:39.105952 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:30:39.106730 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:39.105998 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:30:39.571812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:39.562749 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr"] Oct 31 13:30:39.590023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:39.585762 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:39.605640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:39.598782 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:39.838741 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:39.838688 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:39.866373 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:39.866317 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded" Oct 31 13:30:40.076145 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.076015 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:40.262208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.262157 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" event={"ID":"204317d2-4400-4923-adac-e410dced28b3","Type":"ContainerStarted","Data":"5b708277ce190efd946845937b0874887e185b0b25590dbf88b72c65c91d96c7"} Oct 31 13:30:40.262488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.262463 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" event={"ID":"204317d2-4400-4923-adac-e410dced28b3","Type":"ContainerStarted","Data":"8420821e84f5c916d1230b39d71b5c1ebcbc0009c0daf03375e06f17abc635ae"} Oct 31 13:30:40.289439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.289402 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" event={"ID":"cdde2ce8-b77f-48c8-ab0a-808aab30216c","Type":"ContainerStarted","Data":"c3b567db43cce0858ca4fb0c231d1c4e0da3a7f3a290b0f74748d60404c607e4"} Oct 31 13:30:40.495620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.470313 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:40.511696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.506340 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:30:40.530626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.524856 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:40.608635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.587969 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:40.608635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.588203 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:30:40.667619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.642075 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:40.676617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.653740 5532 patch_prober.go:29] interesting pod/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:30:40.676617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.673286 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 13:30:40.676617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.673793 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:30:40.758673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.719845 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:40.758673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.720709 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:40.758673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.730021 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:40.821625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.775713 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:40.821625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.778448 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" podStartSLOduration=3.7784285730000002 podStartE2EDuration="3.778428573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:40.317905214 +0000 UTC m=+1608.463649044" watchObservedRunningTime="2026-10-31 13:30:40.778428573 +0000 UTC m=+1608.924172402" Oct 31 13:30:40.883427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.883382 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:40.891624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.887637 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:30:40.976288 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:40.976244 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Oct 31 13:30:41.129055 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.128927 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:41.135823 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.135793 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:30:41.383134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.383037 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:41.525765 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.519799 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:41.525765 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.519923 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:30:41.525765 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.520147 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:30:41.525765 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.522418 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:30:41.629622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.604800 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:41.672150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.484192 5532 patch_prober.go:29] interesting pod/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:30:41.672429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.672396 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 13:30:41.693619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.687502 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:41.753620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.744903 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:41.753620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.745170 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:41.753620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.745248 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:30:41.871731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.848526 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:30:41.967811 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:41.962782 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:30:42.069620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:42.002626 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:30:42.069620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:42.003021 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:30:42.140508 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:42.140462 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:42.573210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:42.573164 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:42.573454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:42.573425 5532 patch_prober.go:29] interesting pod/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:30:42.573706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:42.573658 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 13:30:42.718619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:42.582846 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:30:42.846747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:42.767806 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:30:42.882621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:42.878918 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:30:42.978016 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:42.977969 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:42.978351 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:42.978322 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:42.982617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:42.978866 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:30:42.983152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:42.983119 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:30:43.002620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.000702 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:43.006617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.002909 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": context deadline exceeded" Oct 31 13:30:43.006617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.003672 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:43.186910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.179733 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:43.577718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.577059 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:43.645782 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:43.645747 5532 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.643s" Oct 31 13:30:43.650095 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.650067 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:30:43.650404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.650384 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:43.650555 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.650535 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" event={"ID":"cdde2ce8-b77f-48c8-ab0a-808aab30216c","Type":"ContainerStarted","Data":"ec6273d64b0fe913d83a8df7eb74c1550bb5d49ed376e9c131b81b366325c75a"} Oct 31 13:30:43.650806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.650790 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:30:43.650943 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.650931 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:30:43.725634 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.714340 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/health\": context deadline exceeded" Oct 31 13:30:43.842816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.842684 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:30:43.870620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.866777 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:43.890620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.886718 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:30:43.942674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.932498 5532 generic.go:356] "Generic (PLEG): container finished" podID="cdde2ce8-b77f-48c8-ab0a-808aab30216c" containerID="ec6273d64b0fe913d83a8df7eb74c1550bb5d49ed376e9c131b81b366325c75a" exitCode=0 Oct 31 13:30:43.942674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.933562 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" event={"ID":"cdde2ce8-b77f-48c8-ab0a-808aab30216c","Type":"ContainerDied","Data":"ec6273d64b0fe913d83a8df7eb74c1550bb5d49ed376e9c131b81b366325c75a"} Oct 31 13:30:43.942674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.934296 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:43.942674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.941017 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:30:43.987723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.987684 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:30:44.011621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.007709 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:30:44.016194 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.989713 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:30:44.023095 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.989753 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:30:44.023551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.989772 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:30:44.023906 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.014671 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:44.246678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:43.989679 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:44.304476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.203985 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:44.304772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.204066 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:30:44.304951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.304920 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:44.305083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.229424 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:44.636681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.633077 5532 patch_prober.go:29] interesting pod/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:30:44.636681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.633140 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:44.663641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.649692 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:30:44.663641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.654724 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:44.693773 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:44.693705 5532 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdde2ce8_b77f_48c8_ab0a_808aab30216c.slice/crio-ec6273d64b0fe913d83a8df7eb74c1550bb5d49ed376e9c131b81b366325c75a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdde2ce8_b77f_48c8_ab0a_808aab30216c.slice/crio-conmon-ec6273d64b0fe913d83a8df7eb74c1550bb5d49ed376e9c131b81b366325c75a.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:30:44.722278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.722156 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:30:44.722573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.722244 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:44.870648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.849689 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:30:44.899731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:44.887562 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:30:45.039220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:45.039180 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:45.040432 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:45.040400 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:30:45.045641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:45.040467 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:45.045951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:45.040492 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:30:45.077721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:45.040515 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:30:45.079520 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:45.040531 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:30:45.415009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:45.414139 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" podUID="51b75119-a5cd-4e8c-bd06-3725a6bb30a5" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.29:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:45.429998 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:45.429928 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:45.461026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:45.457796 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Oct 31 13:30:45.517748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:45.458017 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:30:45.633672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:45.458114 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 13:30:45.646643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:45.458945 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" podUID="3f952465-49ad-43cf-9598-dd3741e8de2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:45.717698 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:45.712020 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:46.036676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:46.033207 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:30:46.045739 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:46.045687 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:30:46.050125 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:46.050068 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:30:46.163956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:46.158585 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:46.247789 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:46.247742 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:46.341677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:46.295771 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:30:46.341677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:46.300738 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:46.740248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:46.740199 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:30:46.757667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:46.757623 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 13:30:47.083237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.078357 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:47.098280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.098234 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:47.098935 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.098891 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:30:47.116519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.116439 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:47.125114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.077557 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Oct 31 13:30:47.159083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.159044 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:30:47.184236 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:30:47.184199 5532 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.123s" Oct 31 13:30:47.184417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.184273 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:30:47.184417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.184376 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:30:47.187623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.187586 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" Oct 31 13:30:47.195348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.195275 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" event={"ID":"cdde2ce8-b77f-48c8-ab0a-808aab30216c","Type":"ContainerDied","Data":"c3b567db43cce0858ca4fb0c231d1c4e0da3a7f3a290b0f74748d60404c607e4"} Oct 31 13:30:47.195348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.195324 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3b567db43cce0858ca4fb0c231d1c4e0da3a7f3a290b0f74748d60404c607e4" Oct 31 13:30:47.212985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.212955 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:30:47.236734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.236708 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:30:47.298612 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.291441 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdde2ce8-b77f-48c8-ab0a-808aab30216c-kube-api-access-stz24\" (UniqueName: \"kubernetes.io/projected/cdde2ce8-b77f-48c8-ab0a-808aab30216c-kube-api-access-stz24\") pod \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\" (UID: \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\") " Oct 31 13:30:47.325615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.310119 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-xks-ca\") pod \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\" (UID: \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\") " Oct 31 13:30:47.325615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.298948 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:30:47.325615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.310273 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:30:47.325615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.304171 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:30:47.325615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.312808 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-openshift-ca\") pod \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\" (UID: \"cdde2ce8-b77f-48c8-ab0a-808aab30216c\") " Oct 31 13:30:47.325615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.319176 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-openshift-ca" pod "cdde2ce8-b77f-48c8-ab0a-808aab30216c" (UID: "cdde2ce8-b77f-48c8-ab0a-808aab30216c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:30:47.327482 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.327452 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:47.343556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.334136 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-xks-ca" pod "cdde2ce8-b77f-48c8-ab0a-808aab30216c" (UID: "cdde2ce8-b77f-48c8-ab0a-808aab30216c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:30:47.354435 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.354409 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:30:47.366584 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.366546 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:30:47.379957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.379933 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:30:47.381362 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.381329 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdde2ce8-b77f-48c8-ab0a-808aab30216c-kube-api-access-stz24" pod "cdde2ce8-b77f-48c8-ab0a-808aab30216c" (UID: "cdde2ce8-b77f-48c8-ab0a-808aab30216c"). InnerVolumeSpecName "kube-api-access-stz24". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:30:47.414172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.414145 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:30:47.415886 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.415860 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:30:47.448107 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.447071 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stz24\" (UniqueName: \"kubernetes.io/projected/cdde2ce8-b77f-48c8-ab0a-808aab30216c-kube-api-access-stz24\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:47.448107 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:47.447105 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdde2ce8-b77f-48c8-ab0a-808aab30216c-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:30:48.202739 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.202708 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr" Oct 31 13:30:48.705182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.705150 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj"] Oct 31 13:30:48.705931 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.705912 5532 cpu_manager.go:414] "containerMap: removing container" podUID="cdde2ce8-b77f-48c8-ab0a-808aab30216c" containerName="revoke-keys" Oct 31 13:30:48.705931 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.705931 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdde2ce8-b77f-48c8-ab0a-808aab30216c" containerName="revoke-keys" Oct 31 13:30:48.706064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.706054 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdde2ce8-b77f-48c8-ab0a-808aab30216c" containerName="revoke-keys" Oct 31 13:30:48.710017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.709994 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" Oct 31 13:30:48.716739 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.716714 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-fc1bbe2b-dockercfg-bg49w\"" Oct 31 13:30:48.716872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.716801 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-fc1bbe2b\"" Oct 31 13:30:48.725171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.725147 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj"] Oct 31 13:30:48.809704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.809673 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dz55\" (UniqueName: \"kubernetes.io/projected/aa53b017-4e9c-4da6-944a-d880e98f3cda-kube-api-access-4dz55\") pod \"payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj\" (UID: \"aa53b017-4e9c-4da6-944a-d880e98f3cda\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" Oct 31 13:30:48.811255 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.810006 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa53b017-4e9c-4da6-944a-d880e98f3cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj\" (UID: \"aa53b017-4e9c-4da6-944a-d880e98f3cda\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" Oct 31 13:30:48.911692 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.911657 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dz55\" (UniqueName: \"kubernetes.io/projected/aa53b017-4e9c-4da6-944a-d880e98f3cda-kube-api-access-4dz55\") pod \"payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj\" (UID: \"aa53b017-4e9c-4da6-944a-d880e98f3cda\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" Oct 31 13:30:48.911974 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.911951 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa53b017-4e9c-4da6-944a-d880e98f3cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj\" (UID: \"aa53b017-4e9c-4da6-944a-d880e98f3cda\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" Oct 31 13:30:48.912737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.912714 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa53b017-4e9c-4da6-944a-d880e98f3cda-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj\" (UID: \"aa53b017-4e9c-4da6-944a-d880e98f3cda\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" Oct 31 13:30:48.936393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:48.936365 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dz55\" (UniqueName: \"kubernetes.io/projected/aa53b017-4e9c-4da6-944a-d880e98f3cda-kube-api-access-4dz55\") pod \"payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj\" (UID: \"aa53b017-4e9c-4da6-944a-d880e98f3cda\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" Oct 31 13:30:49.035289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:49.035205 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" Oct 31 13:30:49.334690 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:30:49.334628 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa53b017_4e9c_4da6_944a_d880e98f3cda.slice/crio-3aa3c49bf84d51f25eee799d4eef92d0cea64b3cb92ebdf8ece821299cde8ed1 WatchSource:0}: Error finding container 3aa3c49bf84d51f25eee799d4eef92d0cea64b3cb92ebdf8ece821299cde8ed1: Status 404 returned error can't find the container with id 3aa3c49bf84d51f25eee799d4eef92d0cea64b3cb92ebdf8ece821299cde8ed1 Oct 31 13:30:49.352972 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:49.341844 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj"] Oct 31 13:30:50.224093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:50.224059 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" event={"ID":"aa53b017-4e9c-4da6-944a-d880e98f3cda","Type":"ContainerStarted","Data":"4e2b95c488d317dbad573b3b7abf9984c2a7435763cb271367fb4fbf60bc4c73"} Oct 31 13:30:50.224254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:50.224112 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" event={"ID":"aa53b017-4e9c-4da6-944a-d880e98f3cda","Type":"ContainerStarted","Data":"3aa3c49bf84d51f25eee799d4eef92d0cea64b3cb92ebdf8ece821299cde8ed1"} Oct 31 13:30:50.224322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:50.224310 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" Oct 31 13:30:50.252856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:50.252798 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" podStartSLOduration=2.252784164 podStartE2EDuration="2.252784164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:50.247252022 +0000 UTC m=+1618.392995876" watchObservedRunningTime="2026-10-31 13:30:50.252784164 +0000 UTC m=+1618.398528017" Oct 31 13:30:56.867335 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:56.867288 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h"] Oct 31 13:30:56.873723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:56.873702 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" Oct 31 13:30:56.881657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:56.881631 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a66ae960\"" Oct 31 13:30:56.881905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:56.881862 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a66ae960-dockercfg-ql9xg\"" Oct 31 13:30:56.883155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:56.883124 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h"] Oct 31 13:30:56.953142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:56.953108 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h\" (UID: \"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" Oct 31 13:30:56.953314 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:56.953251 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brsfb\" (UniqueName: \"kubernetes.io/projected/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-kube-api-access-brsfb\") pod \"payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h\" (UID: \"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" Oct 31 13:30:57.056588 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.056535 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h\" (UID: \"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" Oct 31 13:30:57.056781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.056665 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brsfb\" (UniqueName: \"kubernetes.io/projected/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-kube-api-access-brsfb\") pod \"payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h\" (UID: \"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" Oct 31 13:30:57.058406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.058376 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h\" (UID: \"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" Oct 31 13:30:57.076669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.076632 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brsfb\" (UniqueName: \"kubernetes.io/projected/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-kube-api-access-brsfb\") pod \"payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h\" (UID: \"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" Oct 31 13:30:57.195561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.195525 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" Oct 31 13:30:57.561466 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:30:57.561430 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f1fc723_b9d7_4b78_a23e_e18df2aa2d5c.slice/crio-e1be558e48db808bae37e4f146cda12bd30d4a057af84190942e85be77598c39 WatchSource:0}: Error finding container e1be558e48db808bae37e4f146cda12bd30d4a057af84190942e85be77598c39: Status 404 returned error can't find the container with id e1be558e48db808bae37e4f146cda12bd30d4a057af84190942e85be77598c39 Oct 31 13:30:57.563500 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.563472 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h"] Oct 31 13:30:57.607691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.607578 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7"] Oct 31 13:30:57.622387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.622359 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:57.623796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.623769 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7"] Oct 31 13:30:57.629932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.629890 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8439b78a-serving-cert\"" Oct 31 13:30:57.704436 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.704407 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-maas-api-tls\") pod \"maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:57.704436 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.704439 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-metrics-certs\") pod \"maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:57.704653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.704558 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n27vv\" (UniqueName: \"kubernetes.io/projected/373a9ad9-0eac-4420-876b-c0ad17235652-kube-api-access-n27vv\") pod \"maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:57.704694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.704662 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373a9ad9-0eac-4420-876b-c0ad17235652-openshift-service-ca\") pod \"maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:57.817706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.815784 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-maas-api-tls\") pod \"maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:57.817706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.815843 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-metrics-certs\") pod \"maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:57.817706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.815944 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n27vv\" (UniqueName: \"kubernetes.io/projected/373a9ad9-0eac-4420-876b-c0ad17235652-kube-api-access-n27vv\") pod \"maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:57.817706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.816008 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373a9ad9-0eac-4420-876b-c0ad17235652-openshift-service-ca\") pod \"maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:57.818168 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.817777 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373a9ad9-0eac-4420-876b-c0ad17235652-openshift-service-ca\") pod \"maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:57.824383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.824350 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-maas-api-tls\") pod \"maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:57.825697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.825667 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-metrics-certs\") pod \"maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:57.834894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.834860 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n27vv\" (UniqueName: \"kubernetes.io/projected/373a9ad9-0eac-4420-876b-c0ad17235652-kube-api-access-n27vv\") pod \"maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:57.956464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:57.956430 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:58.263034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:58.262997 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7"] Oct 31 13:30:58.264772 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:30:58.264739 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373a9ad9_0eac_4420_876b_c0ad17235652.slice/crio-64351ec708799cc8708ddcf1c2c64b5a54a5b9c0a1762af3408fd27a1c871298 WatchSource:0}: Error finding container 64351ec708799cc8708ddcf1c2c64b5a54a5b9c0a1762af3408fd27a1c871298: Status 404 returned error can't find the container with id 64351ec708799cc8708ddcf1c2c64b5a54a5b9c0a1762af3408fd27a1c871298 Oct 31 13:30:58.361906 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:58.361864 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" event={"ID":"373a9ad9-0eac-4420-876b-c0ad17235652","Type":"ContainerStarted","Data":"64351ec708799cc8708ddcf1c2c64b5a54a5b9c0a1762af3408fd27a1c871298"} Oct 31 13:30:58.363899 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:58.363869 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" event={"ID":"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c","Type":"ContainerStarted","Data":"3fb19e25cb94821b7924f2e39699157181ae274b833862de3bf4f3e8b63ed83a"} Oct 31 13:30:58.364019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:58.363915 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" event={"ID":"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c","Type":"ContainerStarted","Data":"e1be558e48db808bae37e4f146cda12bd30d4a057af84190942e85be77598c39"} Oct 31 13:30:58.364972 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:58.364926 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" Oct 31 13:30:58.388874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:58.388815 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" podStartSLOduration=2.388799456 podStartE2EDuration="2.388799456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:58.385811642 +0000 UTC m=+1626.531555469" watchObservedRunningTime="2026-10-31 13:30:58.388799456 +0000 UTC m=+1626.534543284" Oct 31 13:30:59.377545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:59.377499 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" event={"ID":"373a9ad9-0eac-4420-876b-c0ad17235652","Type":"ContainerStarted","Data":"415fdee0899ebab7a6a58ef7a7a33b25a5206fd4abb9bcbcef7bc9a78d0ab26a"} Oct 31 13:30:59.378092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:59.377575 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:30:59.407546 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:30:59.407484 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" podStartSLOduration=1.979604637 podStartE2EDuration="2.40746823s" totalImagesPullingTime="427.863593ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:30:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:30:59.406462761 +0000 UTC m=+1627.552206589" watchObservedRunningTime="2026-10-31 13:30:59.40746823 +0000 UTC m=+1627.553212057" Oct 31 13:31:00.680698 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.671697 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2"] Oct 31 13:31:00.700704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.687941 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:00.700704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.690388 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2"] Oct 31 13:31:00.700704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.695141 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-fc1bbe2b-serving-cert\"" Oct 31 13:31:00.782885 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.782814 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84z4m\" (UniqueName: \"kubernetes.io/projected/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-kube-api-access-84z4m\") pod \"maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:00.783101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.783064 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-openshift-service-ca\") pod \"maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:00.783268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.783246 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-metrics-certs\") pod \"maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:00.783520 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.783496 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-maas-api-tls\") pod \"maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:00.897978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.894204 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-openshift-service-ca\") pod \"maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:00.897978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.894436 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-metrics-certs\") pod \"maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:00.897978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.894504 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-maas-api-tls\") pod \"maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:00.897978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.894577 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84z4m\" (UniqueName: \"kubernetes.io/projected/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-kube-api-access-84z4m\") pod \"maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:00.897978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.896225 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-openshift-service-ca\") pod \"maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:00.900116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.900071 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-metrics-certs\") pod \"maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:00.913712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.912948 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84z4m\" (UniqueName: \"kubernetes.io/projected/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-kube-api-access-84z4m\") pod \"maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:00.916222 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:00.916194 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-maas-api-tls\") pod \"maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:01.050702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:01.050613 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:01.237247 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:01.237215 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" Oct 31 13:31:01.364883 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:31:01.364841 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dd4fba5_acda_4071_bc7d_f9b4ae0a4ca5.slice/crio-be6b59c37531dc5671daa3f29a3f879a612b9c8b8cfc97062c723d13135071cf WatchSource:0}: Error finding container be6b59c37531dc5671daa3f29a3f879a612b9c8b8cfc97062c723d13135071cf: Status 404 returned error can't find the container with id be6b59c37531dc5671daa3f29a3f879a612b9c8b8cfc97062c723d13135071cf Oct 31 13:31:01.365708 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:01.365680 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2"] Oct 31 13:31:01.406338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:01.406303 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" event={"ID":"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5","Type":"ContainerStarted","Data":"be6b59c37531dc5671daa3f29a3f879a612b9c8b8cfc97062c723d13135071cf"} Oct 31 13:31:01.773751 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:31:01.773710 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:31:02.416666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:02.416585 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" event={"ID":"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5","Type":"ContainerStarted","Data":"a82c7b81afc401a2f2b966c9a3efa7d32aa0233d123ab33d55dd026ed0bc5c44"} Oct 31 13:31:02.416848 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:02.416699 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:02.443342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:02.443291 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" podStartSLOduration=1.984359965 podStartE2EDuration="2.443277261s" totalImagesPullingTime="458.917296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:02.439537089 +0000 UTC m=+1630.585280916" watchObservedRunningTime="2026-10-31 13:31:02.443277261 +0000 UTC m=+1630.589021152" Oct 31 13:31:05.399001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:05.398971 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:31:05.944241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:05.944210 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68"] Oct 31 13:31:05.949118 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:05.949101 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" Oct 31 13:31:05.962081 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:05.962058 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a8c1600c-dockercfg-cpf5c\"" Oct 31 13:31:05.967186 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:05.967161 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68"] Oct 31 13:31:05.999861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:05.999833 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a8c1600c\"" Oct 31 13:31:06.104176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.104139 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddksq\" (UniqueName: \"kubernetes.io/projected/5eb5abfa-b925-49ee-8b96-383bb328b6d0-kube-api-access-ddksq\") pod \"payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68\" (UID: \"5eb5abfa-b925-49ee-8b96-383bb328b6d0\") " pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" Oct 31 13:31:06.104384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.104301 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb5abfa-b925-49ee-8b96-383bb328b6d0-plugins-config-volume\") pod \"payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68\" (UID: \"5eb5abfa-b925-49ee-8b96-383bb328b6d0\") " pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" Oct 31 13:31:06.208631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.208113 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddksq\" (UniqueName: \"kubernetes.io/projected/5eb5abfa-b925-49ee-8b96-383bb328b6d0-kube-api-access-ddksq\") pod \"payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68\" (UID: \"5eb5abfa-b925-49ee-8b96-383bb328b6d0\") " pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" Oct 31 13:31:06.208631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.208245 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb5abfa-b925-49ee-8b96-383bb328b6d0-plugins-config-volume\") pod \"payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68\" (UID: \"5eb5abfa-b925-49ee-8b96-383bb328b6d0\") " pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" Oct 31 13:31:06.213667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.209350 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb5abfa-b925-49ee-8b96-383bb328b6d0-plugins-config-volume\") pod \"payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68\" (UID: \"5eb5abfa-b925-49ee-8b96-383bb328b6d0\") " pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" Oct 31 13:31:06.228015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.227980 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddksq\" (UniqueName: \"kubernetes.io/projected/5eb5abfa-b925-49ee-8b96-383bb328b6d0-kube-api-access-ddksq\") pod \"payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68\" (UID: \"5eb5abfa-b925-49ee-8b96-383bb328b6d0\") " pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" Oct 31 13:31:06.273089 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.272993 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" Oct 31 13:31:06.597249 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:31:06.597212 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb5abfa_b925_49ee_8b96_383bb328b6d0.slice/crio-116970d5f8cf942cf70b539194f0c5b9c4660b5c8ba97d50cdeda662406166fc WatchSource:0}: Error finding container 116970d5f8cf942cf70b539194f0c5b9c4660b5c8ba97d50cdeda662406166fc: Status 404 returned error can't find the container with id 116970d5f8cf942cf70b539194f0c5b9c4660b5c8ba97d50cdeda662406166fc Oct 31 13:31:06.598653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.598625 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68"] Oct 31 13:31:06.865435 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.865362 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b"] Oct 31 13:31:06.870055 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.870035 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" Oct 31 13:31:06.879013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.878984 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-dff70365\"" Oct 31 13:31:06.883163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.883129 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b"] Oct 31 13:31:06.886833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.886810 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-dff70365-dockercfg-j9k52\"" Oct 31 13:31:06.926026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.925992 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664"] Oct 31 13:31:06.941344 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.941032 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" Oct 31 13:31:06.949437 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:06.949408 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664"] Oct 31 13:31:07.021201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.021166 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f170a569-5b9f-4258-8456-6f3e195c4d46-plugins-config-volume\") pod \"payload-processing-e2e-ait-dff70365-577bb65985-lv664\" (UID: \"f170a569-5b9f-4258-8456-6f3e195c4d46\") " pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" Oct 31 13:31:07.021201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.021207 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtrff\" (UniqueName: \"kubernetes.io/projected/42886c31-8e0b-44e7-b1da-f3cddb479e92-kube-api-access-qtrff\") pod \"payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b\" (UID: \"42886c31-8e0b-44e7-b1da-f3cddb479e92\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" Oct 31 13:31:07.021422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.021365 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rplq\" (UniqueName: \"kubernetes.io/projected/f170a569-5b9f-4258-8456-6f3e195c4d46-kube-api-access-5rplq\") pod \"payload-processing-e2e-ait-dff70365-577bb65985-lv664\" (UID: \"f170a569-5b9f-4258-8456-6f3e195c4d46\") " pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" Oct 31 13:31:07.021487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.021470 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42886c31-8e0b-44e7-b1da-f3cddb479e92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b\" (UID: \"42886c31-8e0b-44e7-b1da-f3cddb479e92\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" Oct 31 13:31:07.141976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.141874 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f170a569-5b9f-4258-8456-6f3e195c4d46-plugins-config-volume\") pod \"payload-processing-e2e-ait-dff70365-577bb65985-lv664\" (UID: \"f170a569-5b9f-4258-8456-6f3e195c4d46\") " pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" Oct 31 13:31:07.143083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.143049 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtrff\" (UniqueName: \"kubernetes.io/projected/42886c31-8e0b-44e7-b1da-f3cddb479e92-kube-api-access-qtrff\") pod \"payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b\" (UID: \"42886c31-8e0b-44e7-b1da-f3cddb479e92\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" Oct 31 13:31:07.143411 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.143388 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rplq\" (UniqueName: \"kubernetes.io/projected/f170a569-5b9f-4258-8456-6f3e195c4d46-kube-api-access-5rplq\") pod \"payload-processing-e2e-ait-dff70365-577bb65985-lv664\" (UID: \"f170a569-5b9f-4258-8456-6f3e195c4d46\") " pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" Oct 31 13:31:07.143675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.143655 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42886c31-8e0b-44e7-b1da-f3cddb479e92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b\" (UID: \"42886c31-8e0b-44e7-b1da-f3cddb479e92\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" Oct 31 13:31:07.143978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.143461 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f170a569-5b9f-4258-8456-6f3e195c4d46-plugins-config-volume\") pod \"payload-processing-e2e-ait-dff70365-577bb65985-lv664\" (UID: \"f170a569-5b9f-4258-8456-6f3e195c4d46\") " pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" Oct 31 13:31:07.152619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.144932 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42886c31-8e0b-44e7-b1da-f3cddb479e92-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b\" (UID: \"42886c31-8e0b-44e7-b1da-f3cddb479e92\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" Oct 31 13:31:07.183617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.175267 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtrff\" (UniqueName: \"kubernetes.io/projected/42886c31-8e0b-44e7-b1da-f3cddb479e92-kube-api-access-qtrff\") pod \"payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b\" (UID: \"42886c31-8e0b-44e7-b1da-f3cddb479e92\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" Oct 31 13:31:07.183617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.175957 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rplq\" (UniqueName: \"kubernetes.io/projected/f170a569-5b9f-4258-8456-6f3e195c4d46-kube-api-access-5rplq\") pod \"payload-processing-e2e-ait-dff70365-577bb65985-lv664\" (UID: \"f170a569-5b9f-4258-8456-6f3e195c4d46\") " pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" Oct 31 13:31:07.193618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.188704 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" Oct 31 13:31:07.264618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.263576 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" Oct 31 13:31:07.493487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.493442 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" event={"ID":"5eb5abfa-b925-49ee-8b96-383bb328b6d0","Type":"ContainerStarted","Data":"1d21ff5f20085f1bddb6ecd3f5cb792947772bb88c68419e6172a7dd7e112da4"} Oct 31 13:31:07.493952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.493516 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" event={"ID":"5eb5abfa-b925-49ee-8b96-383bb328b6d0","Type":"ContainerStarted","Data":"116970d5f8cf942cf70b539194f0c5b9c4660b5c8ba97d50cdeda662406166fc"} Oct 31 13:31:07.493952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.493823 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" Oct 31 13:31:07.531198 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.523404 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b"] Oct 31 13:31:07.531198 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.527411 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" podStartSLOduration=2.527391334 podStartE2EDuration="2.527391334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:07.523787657 +0000 UTC m=+1635.669531485" watchObservedRunningTime="2026-10-31 13:31:07.527391334 +0000 UTC m=+1635.673135166" Oct 31 13:31:07.599248 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:31:07.599221 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf170a569_5b9f_4258_8456_6f3e195c4d46.slice/crio-6f10ea934cc23bee6bf5a1c3b554d02d2e43d76e514b242ea8239f92b0b2f6d8 WatchSource:0}: Error finding container 6f10ea934cc23bee6bf5a1c3b554d02d2e43d76e514b242ea8239f92b0b2f6d8: Status 404 returned error can't find the container with id 6f10ea934cc23bee6bf5a1c3b554d02d2e43d76e514b242ea8239f92b0b2f6d8 Oct 31 13:31:07.600941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:07.600918 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664"] Oct 31 13:31:08.434096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:08.434069 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:08.514382 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:08.514328 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" event={"ID":"f170a569-5b9f-4258-8456-6f3e195c4d46","Type":"ContainerStarted","Data":"5f0f072eb3b53f65bb861e6c44957a02aa0fde8689fc08dd56301dc589e723cc"} Oct 31 13:31:08.515303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:08.515191 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" Oct 31 13:31:08.515303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:08.515228 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" event={"ID":"f170a569-5b9f-4258-8456-6f3e195c4d46","Type":"ContainerStarted","Data":"6f10ea934cc23bee6bf5a1c3b554d02d2e43d76e514b242ea8239f92b0b2f6d8"} Oct 31 13:31:08.519409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:08.519368 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" event={"ID":"42886c31-8e0b-44e7-b1da-f3cddb479e92","Type":"ContainerStarted","Data":"38beb8b6dfff6f0b354ccbb00727a710d91e30e95b4f0cb192cc62a05944b1e9"} Oct 31 13:31:08.519585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:08.519423 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" event={"ID":"42886c31-8e0b-44e7-b1da-f3cddb479e92","Type":"ContainerStarted","Data":"84d2af85c37012b741e421c5e9837f4a25ee691712d952995fd53468aceb4c76"} Oct 31 13:31:08.519585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:08.519453 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" Oct 31 13:31:08.580057 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:08.579994 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" podStartSLOduration=2.579975097 podStartE2EDuration="2.579975097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:08.567947278 +0000 UTC m=+1636.713691103" watchObservedRunningTime="2026-10-31 13:31:08.579975097 +0000 UTC m=+1636.725718926" Oct 31 13:31:08.603707 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:08.603647 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" podStartSLOduration=2.60362927 podStartE2EDuration="2.60362927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:08.600050719 +0000 UTC m=+1636.745794548" watchObservedRunningTime="2026-10-31 13:31:08.60362927 +0000 UTC m=+1636.749373100" Oct 31 13:31:10.397608 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:10.397553 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" Oct 31 13:31:18.361619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.354162 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd"] Oct 31 13:31:18.385616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.375499 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" Oct 31 13:31:18.406618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.400858 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:31:18.406618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.401855 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:31:18.422044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.413282 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd"] Oct 31 13:31:18.561578 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.561547 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" Oct 31 13:31:18.666354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.666315 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd\" (UID: \"c56f93a4-4c00-4b39-b175-952105ec043d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" Oct 31 13:31:18.764785 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.764745 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkgnf\" (UniqueName: \"kubernetes.io/projected/c56f93a4-4c00-4b39-b175-952105ec043d-kube-api-access-hkgnf\") pod \"maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd\" (UID: \"c56f93a4-4c00-4b39-b175-952105ec043d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" Oct 31 13:31:18.765120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.765100 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd\" (UID: \"c56f93a4-4c00-4b39-b175-952105ec043d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" Oct 31 13:31:18.946142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.923670 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd\" (UID: \"c56f93a4-4c00-4b39-b175-952105ec043d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" Oct 31 13:31:18.946142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.923864 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkgnf\" (UniqueName: \"kubernetes.io/projected/c56f93a4-4c00-4b39-b175-952105ec043d-kube-api-access-hkgnf\") pod \"maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd\" (UID: \"c56f93a4-4c00-4b39-b175-952105ec043d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" Oct 31 13:31:18.946142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.923966 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd\" (UID: \"c56f93a4-4c00-4b39-b175-952105ec043d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" Oct 31 13:31:18.946142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.924929 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd\" (UID: \"c56f93a4-4c00-4b39-b175-952105ec043d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" Oct 31 13:31:19.022620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:18.982783 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd\" (UID: \"c56f93a4-4c00-4b39-b175-952105ec043d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" Oct 31 13:31:19.117900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:19.117861 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkgnf\" (UniqueName: \"kubernetes.io/projected/c56f93a4-4c00-4b39-b175-952105ec043d-kube-api-access-hkgnf\") pod \"maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd\" (UID: \"c56f93a4-4c00-4b39-b175-952105ec043d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" Oct 31 13:31:19.385672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:19.370109 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" Oct 31 13:31:19.624899 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:19.624854 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" Oct 31 13:31:19.673462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:19.673415 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" Oct 31 13:31:19.740329 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:19.724812 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:19.875176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:19.875047 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded" Oct 31 13:31:19.897225 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:19.894827 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:19.919744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:19.919701 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:31:19.985039 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:19.983749 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:20.041619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.041177 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:31:20.066667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.046800 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:31:20.071089 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.071046 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:31:20.361865 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.361758 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:20.374777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.362671 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:20.407940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.368869 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:20.421712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.421665 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:31:20.770619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.742920 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:20.770619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.743544 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:31:20.833312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.833279 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:31:20.913248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.913201 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:20.915122 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.915095 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:31:20.932032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.923742 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:20.932032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.924301 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:31:20.994783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.972428 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:21.007296 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:20.986275 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:21.008157 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.008133 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:31:21.100077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.095885 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:21.100077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.096325 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:31:21.104627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.101332 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:31:21.104627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.101683 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:21.104627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.101983 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:31:21.104627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.102039 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:31:21.386674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.375421 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 13:31:21.386674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.375950 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:31:21.421888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.421845 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:21.429625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.426980 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:31:21.429625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.428073 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:31:21.429625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.428713 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:31:21.429625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.428769 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:31:21.429625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.428804 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:31:21.979621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:21.969060 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:22.013615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:22.007791 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:31:22.186685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:22.085162 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" podUID="51b75119-a5cd-4e8c-bd06-3725a6bb30a5" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.29:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:22.186685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:22.117813 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:22.186685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:22.121517 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:31:22.186685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:22.121790 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:22.186685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:22.122288 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:31:22.437332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:22.437223 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:22.438659 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:22.438573 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:22.445376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:22.445099 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 13:31:22.493108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:22.493065 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:31:23.013306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:22.969998 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:23.088239 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.066113 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:31:23.117642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.097811 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:23.120042 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.119321 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:23.120042 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.119417 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:31:23.137648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.125036 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:23.137648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.125320 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:31:23.162302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.153799 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:23.328917 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:31:23.328836 5532 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.198s" Oct 31 13:31:23.329211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.329193 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:31:23.329342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.329332 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:31:23.448644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.433991 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" event={"ID":"c56f93a4-4c00-4b39-b175-952105ec043d","Type":"ContainerStarted","Data":"c91196cf744eddf461a80a7ae340fb6ffa38281e8005c7cba0df639ccb8ac529"} Oct 31 13:31:23.448644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.444927 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:23.448644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.445527 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:23.516905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.506614 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:23.628404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.628376 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:31:23.719619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.719518 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:31:23.723968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.721398 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:31:23.723968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.723154 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:31:23.732125 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.732106 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:31:23.795625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.786759 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:31:23.837624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.837574 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd"] Oct 31 13:31:23.903567 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.903539 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:31:23.906884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.906859 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:31:23.925142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.925120 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw"] Oct 31 13:31:23.934583 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.934557 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw"] Oct 31 13:31:23.934829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.934736 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:23.993039 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:23.993007 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a8c1600c-serving-cert\"" Oct 31 13:31:24.035822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.035793 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-metrics-certs\") pod \"maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:24.035961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.035871 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b9630f-8938-4110-b18b-c7591eaab085-openshift-service-ca\") pod \"maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:24.035961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.035924 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24fjn\" (UniqueName: \"kubernetes.io/projected/47b9630f-8938-4110-b18b-c7591eaab085-kube-api-access-24fjn\") pod \"maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:24.036054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.036038 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-maas-api-tls\") pod \"maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:24.138734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.137080 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b9630f-8938-4110-b18b-c7591eaab085-openshift-service-ca\") pod \"maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:24.138734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.137137 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fjn\" (UniqueName: \"kubernetes.io/projected/47b9630f-8938-4110-b18b-c7591eaab085-kube-api-access-24fjn\") pod \"maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:24.138734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.137208 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-maas-api-tls\") pod \"maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:24.138734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.137328 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-metrics-certs\") pod \"maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:24.138734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.137838 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b9630f-8938-4110-b18b-c7591eaab085-openshift-service-ca\") pod \"maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:24.185129 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.185014 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-maas-api-tls\") pod \"maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:24.194585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.194517 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fjn\" (UniqueName: \"kubernetes.io/projected/47b9630f-8938-4110-b18b-c7591eaab085-kube-api-access-24fjn\") pod \"maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:24.203782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.203748 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-metrics-certs\") pod \"maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:24.337337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.337305 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:24.716506 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.716457 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" event={"ID":"c56f93a4-4c00-4b39-b175-952105ec043d","Type":"ContainerStarted","Data":"12c577584e720b4b6a25e716b5c64ebb401d4d73e7e1a10fa40626e69e878379"} Oct 31 13:31:24.827556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:24.827489 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" podStartSLOduration=6.827469319 podStartE2EDuration="6.827469319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:24.788740627 +0000 UTC m=+1652.934484455" watchObservedRunningTime="2026-10-31 13:31:24.827469319 +0000 UTC m=+1652.973213150" Oct 31 13:31:25.309068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.309032 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw"] Oct 31 13:31:25.416696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.414479 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb"] Oct 31 13:31:25.427038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.427011 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb"] Oct 31 13:31:25.427332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.427127 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.434248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.434214 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-dff70365-serving-cert\"" Oct 31 13:31:25.438513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.438449 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-maas-api-tls\") pod \"maas-api-e2e-ait-dff70365-5745fb6488-8pmzb\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.438666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.438566 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kjw7\" (UniqueName: \"kubernetes.io/projected/5fe27f22-54e9-4332-aedc-3092b826b153-kube-api-access-2kjw7\") pod \"maas-api-e2e-ait-dff70365-5745fb6488-8pmzb\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.458960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.438762 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe27f22-54e9-4332-aedc-3092b826b153-openshift-service-ca\") pod \"maas-api-e2e-ait-dff70365-5745fb6488-8pmzb\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.458960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.438989 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-metrics-certs\") pod \"maas-api-e2e-ait-dff70365-5745fb6488-8pmzb\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.558689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.548743 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kjw7\" (UniqueName: \"kubernetes.io/projected/5fe27f22-54e9-4332-aedc-3092b826b153-kube-api-access-2kjw7\") pod \"maas-api-e2e-ait-dff70365-5745fb6488-8pmzb\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.570958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.570907 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe27f22-54e9-4332-aedc-3092b826b153-openshift-service-ca\") pod \"maas-api-e2e-ait-dff70365-5745fb6488-8pmzb\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.597996 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.597958 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-metrics-certs\") pod \"maas-api-e2e-ait-dff70365-5745fb6488-8pmzb\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.598875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.598847 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-maas-api-tls\") pod \"maas-api-e2e-ait-dff70365-5745fb6488-8pmzb\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.634769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.576482 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe27f22-54e9-4332-aedc-3092b826b153-openshift-service-ca\") pod \"maas-api-e2e-ait-dff70365-5745fb6488-8pmzb\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.655696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.635487 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-metrics-certs\") pod \"maas-api-e2e-ait-dff70365-5745fb6488-8pmzb\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.655904 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.653911 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-maas-api-tls\") pod \"maas-api-e2e-ait-dff70365-5745fb6488-8pmzb\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.675678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.668829 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kjw7\" (UniqueName: \"kubernetes.io/projected/5fe27f22-54e9-4332-aedc-3092b826b153-kube-api-access-2kjw7\") pod \"maas-api-e2e-ait-dff70365-5745fb6488-8pmzb\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.827020 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.826920 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:25.913517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:25.913465 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" event={"ID":"47b9630f-8938-4110-b18b-c7591eaab085","Type":"ContainerStarted","Data":"62f59507b6d2282bee44134536fe626098e98f4c3897a3db8d9b1805d4b84e07"} Oct 31 13:31:26.045928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:26.045809 5532 generic.go:356] "Generic (PLEG): container finished" podID="c56f93a4-4c00-4b39-b175-952105ec043d" containerID="12c577584e720b4b6a25e716b5c64ebb401d4d73e7e1a10fa40626e69e878379" exitCode=0 Oct 31 13:31:26.122812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:26.122732 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" event={"ID":"c56f93a4-4c00-4b39-b175-952105ec043d","Type":"ContainerDied","Data":"12c577584e720b4b6a25e716b5c64ebb401d4d73e7e1a10fa40626e69e878379"} Oct 31 13:31:26.536411 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:31:26.536381 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fe27f22_54e9_4332_aedc_3092b826b153.slice/crio-d4cab0507f8b06ec8dea1047fc846422c1ee80482aa02baa7b801a71f6cf8599 WatchSource:0}: Error finding container d4cab0507f8b06ec8dea1047fc846422c1ee80482aa02baa7b801a71f6cf8599: Status 404 returned error can't find the container with id d4cab0507f8b06ec8dea1047fc846422c1ee80482aa02baa7b801a71f6cf8599 Oct 31 13:31:26.536528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:26.536504 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb"] Oct 31 13:31:27.057797 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.057766 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" event={"ID":"47b9630f-8938-4110-b18b-c7591eaab085","Type":"ContainerStarted","Data":"0cc0edede1823b21b1dbae33dbddcb339bea9d38d0a40266cc20c05a350284b6"} Oct 31 13:31:27.058248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.057870 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:27.059786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.059755 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" event={"ID":"5fe27f22-54e9-4332-aedc-3092b826b153","Type":"ContainerStarted","Data":"d4cab0507f8b06ec8dea1047fc846422c1ee80482aa02baa7b801a71f6cf8599"} Oct 31 13:31:27.086842 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.086791 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" podStartSLOduration=3.181656276 podStartE2EDuration="4.086775949s" totalImagesPullingTime="905.119673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:27.084976279 +0000 UTC m=+1655.230720107" watchObservedRunningTime="2026-10-31 13:31:27.086775949 +0000 UTC m=+1655.232519777" Oct 31 13:31:27.348920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.348895 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" Oct 31 13:31:27.374440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.373957 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-openshift-ca\") pod \"c56f93a4-4c00-4b39-b175-952105ec043d\" (UID: \"c56f93a4-4c00-4b39-b175-952105ec043d\") " Oct 31 13:31:27.374440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.374115 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-xks-ca\") pod \"c56f93a4-4c00-4b39-b175-952105ec043d\" (UID: \"c56f93a4-4c00-4b39-b175-952105ec043d\") " Oct 31 13:31:27.374690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.374567 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-openshift-ca" pod "c56f93a4-4c00-4b39-b175-952105ec043d" (UID: "c56f93a4-4c00-4b39-b175-952105ec043d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:27.374767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.374706 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c56f93a4-4c00-4b39-b175-952105ec043d-kube-api-access-hkgnf\" (UniqueName: \"kubernetes.io/projected/c56f93a4-4c00-4b39-b175-952105ec043d-kube-api-access-hkgnf\") pod \"c56f93a4-4c00-4b39-b175-952105ec043d\" (UID: \"c56f93a4-4c00-4b39-b175-952105ec043d\") " Oct 31 13:31:27.375570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.375537 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:27.378085 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.378042 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c56f93a4-4c00-4b39-b175-952105ec043d-kube-api-access-hkgnf" pod "c56f93a4-4c00-4b39-b175-952105ec043d" (UID: "c56f93a4-4c00-4b39-b175-952105ec043d"). InnerVolumeSpecName "kube-api-access-hkgnf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:27.383715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.383684 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-xks-ca" pod "c56f93a4-4c00-4b39-b175-952105ec043d" (UID: "c56f93a4-4c00-4b39-b175-952105ec043d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:27.477100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.477067 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c56f93a4-4c00-4b39-b175-952105ec043d-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:27.477254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:27.477102 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkgnf\" (UniqueName: \"kubernetes.io/projected/c56f93a4-4c00-4b39-b175-952105ec043d-kube-api-access-hkgnf\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:28.076078 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:28.076042 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" event={"ID":"5fe27f22-54e9-4332-aedc-3092b826b153","Type":"ContainerStarted","Data":"13d8b206b514dfec9705b9bd2b69fd80cbce25c3489bce7a79c954ac463d14e5"} Oct 31 13:31:28.076577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:28.076557 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:28.077781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:28.077762 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" Oct 31 13:31:28.077884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:28.077801 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd" event={"ID":"c56f93a4-4c00-4b39-b175-952105ec043d","Type":"ContainerDied","Data":"c91196cf744eddf461a80a7ae340fb6ffa38281e8005c7cba0df639ccb8ac529"} Oct 31 13:31:28.077884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:28.077841 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c91196cf744eddf461a80a7ae340fb6ffa38281e8005c7cba0df639ccb8ac529" Oct 31 13:31:28.102887 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:28.102831 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" podStartSLOduration=2.534105349 podStartE2EDuration="3.102812856s" totalImagesPullingTime="568.707507ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:28.097683505 +0000 UTC m=+1656.243427333" watchObservedRunningTime="2026-10-31 13:31:28.102812856 +0000 UTC m=+1656.248556686" Oct 31 13:31:28.491121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:28.491087 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj"] Oct 31 13:31:29.976746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:29.976707 5532 patch_prober.go:29] interesting pod/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:31:29.977347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:29.976764 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:31.769554 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:31:31.769510 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:31:32.278030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:32.277930 5532 patch_prober.go:29] interesting pod/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": read tcp 10.132.0.2:46396->10.132.0.112:15021: read: connection reset by peer" start-of-body= Oct 31 13:31:32.278162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:32.278102 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": read tcp 10.132.0.2:46396->10.132.0.112:15021: read: connection reset by peer" Oct 31 13:31:33.082845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.082815 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:31:33.241688 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.241646 5532 generic.go:356] "Generic (PLEG): container finished" podID="255e3805-e523-4620-b06a-f4d8fa91538f" containerID="b5c85baeaf8e3bc4d53903705543dfe2296d252cd396456529c5676604641255" exitCode=0 Oct 31 13:31:33.241861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.241725 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" event={"ID":"255e3805-e523-4620-b06a-f4d8fa91538f","Type":"ContainerDied","Data":"b5c85baeaf8e3bc4d53903705543dfe2296d252cd396456529c5676604641255"} Oct 31 13:31:33.242310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.242291 5532 scope.go:122] "RemoveContainer" containerID="b5c85baeaf8e3bc4d53903705543dfe2296d252cd396456529c5676604641255" Oct 31 13:31:33.243725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.243706 5532 generic.go:356] "Generic (PLEG): container finished" podID="204317d2-4400-4923-adac-e410dced28b3" containerID="5b708277ce190efd946845937b0874887e185b0b25590dbf88b72c65c91d96c7" exitCode=0 Oct 31 13:31:33.243797 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.243774 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" event={"ID":"204317d2-4400-4923-adac-e410dced28b3","Type":"ContainerDied","Data":"5b708277ce190efd946845937b0874887e185b0b25590dbf88b72c65c91d96c7"} Oct 31 13:31:33.244166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.244152 5532 scope.go:122] "RemoveContainer" containerID="5b708277ce190efd946845937b0874887e185b0b25590dbf88b72c65c91d96c7" Oct 31 13:31:33.244202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.244173 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:31:33.244249 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.244236 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:31:33.244283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.244273 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:31:33.245754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.245728 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:31:33.245830 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.245788 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:31:33.245830 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.245821 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:31:33.365300 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.365265 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7"] Oct 31 13:31:33.365533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.365503 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" podUID="373a9ad9-0eac-4420-876b-c0ad17235652" containerName="maas-api" containerID="cri-o://415fdee0899ebab7a6a58ef7a7a33b25a5206fd4abb9bcbcef7bc9a78d0ab26a" gracePeriod=30 Oct 31 13:31:33.492124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.492083 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" podUID="aa53b017-4e9c-4da6-944a-d880e98f3cda" containerName="payload-pre-processing" containerID="cri-o://4e2b95c488d317dbad573b3b7abf9984c2a7435763cb271367fb4fbf60bc4c73" gracePeriod=25 Oct 31 13:31:33.946700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:33.946604 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:31:34.072398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.067276 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373a9ad9-0eac-4420-876b-c0ad17235652-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373a9ad9-0eac-4420-876b-c0ad17235652-openshift-service-ca\") pod \"373a9ad9-0eac-4420-876b-c0ad17235652\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " Oct 31 13:31:34.072398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.067510 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-maas-api-tls\") pod \"373a9ad9-0eac-4420-876b-c0ad17235652\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " Oct 31 13:31:34.072398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.067635 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-metrics-certs\") pod \"373a9ad9-0eac-4420-876b-c0ad17235652\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " Oct 31 13:31:34.072398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.067838 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373a9ad9-0eac-4420-876b-c0ad17235652-kube-api-access-n27vv\" (UniqueName: \"kubernetes.io/projected/373a9ad9-0eac-4420-876b-c0ad17235652-kube-api-access-n27vv\") pod \"373a9ad9-0eac-4420-876b-c0ad17235652\" (UID: \"373a9ad9-0eac-4420-876b-c0ad17235652\") " Oct 31 13:31:34.072398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.070262 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373a9ad9-0eac-4420-876b-c0ad17235652-openshift-service-ca" pod "373a9ad9-0eac-4420-876b-c0ad17235652" (UID: "373a9ad9-0eac-4420-876b-c0ad17235652"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:34.092246 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.092213 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373a9ad9-0eac-4420-876b-c0ad17235652-kube-api-access-n27vv" pod "373a9ad9-0eac-4420-876b-c0ad17235652" (UID: "373a9ad9-0eac-4420-876b-c0ad17235652"). InnerVolumeSpecName "kube-api-access-n27vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:34.095954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.095855 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-maas-api-tls" pod "373a9ad9-0eac-4420-876b-c0ad17235652" (UID: "373a9ad9-0eac-4420-876b-c0ad17235652"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:34.097568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.097537 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-metrics-certs" pod "373a9ad9-0eac-4420-876b-c0ad17235652" (UID: "373a9ad9-0eac-4420-876b-c0ad17235652"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:34.181625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.173074 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n27vv\" (UniqueName: \"kubernetes.io/projected/373a9ad9-0eac-4420-876b-c0ad17235652-kube-api-access-n27vv\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.231262 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.231231 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373a9ad9-0eac-4420-876b-c0ad17235652-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.231262 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.231261 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.231588 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.231274 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a9ad9-0eac-4420-876b-c0ad17235652-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.247355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.245936 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:31:34.264392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.264356 5532 generic.go:356] "Generic (PLEG): container finished" podID="aa53b017-4e9c-4da6-944a-d880e98f3cda" containerID="4e2b95c488d317dbad573b3b7abf9984c2a7435763cb271367fb4fbf60bc4c73" exitCode=0 Oct 31 13:31:34.264532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.264458 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" event={"ID":"aa53b017-4e9c-4da6-944a-d880e98f3cda","Type":"ContainerDied","Data":"4e2b95c488d317dbad573b3b7abf9984c2a7435763cb271367fb4fbf60bc4c73"} Oct 31 13:31:34.266517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.266488 5532 generic.go:356] "Generic (PLEG): container finished" podID="373a9ad9-0eac-4420-876b-c0ad17235652" containerID="415fdee0899ebab7a6a58ef7a7a33b25a5206fd4abb9bcbcef7bc9a78d0ab26a" exitCode=0 Oct 31 13:31:34.266711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.266691 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" event={"ID":"373a9ad9-0eac-4420-876b-c0ad17235652","Type":"ContainerDied","Data":"415fdee0899ebab7a6a58ef7a7a33b25a5206fd4abb9bcbcef7bc9a78d0ab26a"} Oct 31 13:31:34.266794 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.266730 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" Oct 31 13:31:34.266794 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.266761 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7" event={"ID":"373a9ad9-0eac-4420-876b-c0ad17235652","Type":"ContainerDied","Data":"64351ec708799cc8708ddcf1c2c64b5a54a5b9c0a1762af3408fd27a1c871298"} Oct 31 13:31:34.266794 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.266793 5532 scope.go:122] "RemoveContainer" containerID="415fdee0899ebab7a6a58ef7a7a33b25a5206fd4abb9bcbcef7bc9a78d0ab26a" Oct 31 13:31:34.273191 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.273162 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" event={"ID":"204317d2-4400-4923-adac-e410dced28b3","Type":"ContainerStarted","Data":"d27b17c4da64a6fb4758670e3fa2df1d9240dfa7759aa15ea4ccb9fc10525bb8"} Oct 31 13:31:34.307984 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.307952 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" event={"ID":"255e3805-e523-4620-b06a-f4d8fa91538f","Type":"ContainerStarted","Data":"53bfa6de10ea0312ccab1eed08465909bb25e18e61442c6bc856c69359ac35bc"} Oct 31 13:31:34.312621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.311224 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" Oct 31 13:31:34.345570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.345535 5532 scope.go:122] "RemoveContainer" containerID="415fdee0899ebab7a6a58ef7a7a33b25a5206fd4abb9bcbcef7bc9a78d0ab26a" Oct 31 13:31:34.347763 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:31:34.346523 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415fdee0899ebab7a6a58ef7a7a33b25a5206fd4abb9bcbcef7bc9a78d0ab26a\": container with ID starting with 415fdee0899ebab7a6a58ef7a7a33b25a5206fd4abb9bcbcef7bc9a78d0ab26a not found: ID does not exist" containerID="415fdee0899ebab7a6a58ef7a7a33b25a5206fd4abb9bcbcef7bc9a78d0ab26a" Oct 31 13:31:34.347763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.346561 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415fdee0899ebab7a6a58ef7a7a33b25a5206fd4abb9bcbcef7bc9a78d0ab26a"} err="failed to get container status \"415fdee0899ebab7a6a58ef7a7a33b25a5206fd4abb9bcbcef7bc9a78d0ab26a\": rpc error: code = NotFound desc = could not find container \"415fdee0899ebab7a6a58ef7a7a33b25a5206fd4abb9bcbcef7bc9a78d0ab26a\": container with ID starting with 415fdee0899ebab7a6a58ef7a7a33b25a5206fd4abb9bcbcef7bc9a78d0ab26a not found: ID does not exist" Oct 31 13:31:34.348567 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.348350 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:31:34.348567 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.348450 5532 patch_prober.go:29] interesting pod/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:31:34.348567 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.348499 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 13:31:34.451706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.447166 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa53b017-4e9c-4da6-944a-d880e98f3cda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa53b017-4e9c-4da6-944a-d880e98f3cda-plugins-config-volume\") pod \"aa53b017-4e9c-4da6-944a-d880e98f3cda\" (UID: \"aa53b017-4e9c-4da6-944a-d880e98f3cda\") " Oct 31 13:31:34.451706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.447379 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa53b017-4e9c-4da6-944a-d880e98f3cda-kube-api-access-4dz55\" (UniqueName: \"kubernetes.io/projected/aa53b017-4e9c-4da6-944a-d880e98f3cda-kube-api-access-4dz55\") pod \"aa53b017-4e9c-4da6-944a-d880e98f3cda\" (UID: \"aa53b017-4e9c-4da6-944a-d880e98f3cda\") " Oct 31 13:31:34.456693 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.456657 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa53b017-4e9c-4da6-944a-d880e98f3cda-plugins-config-volume" pod "aa53b017-4e9c-4da6-944a-d880e98f3cda" (UID: "aa53b017-4e9c-4da6-944a-d880e98f3cda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:34.462657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.460701 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa53b017-4e9c-4da6-944a-d880e98f3cda-kube-api-access-4dz55" pod "aa53b017-4e9c-4da6-944a-d880e98f3cda" (UID: "aa53b017-4e9c-4da6-944a-d880e98f3cda"). InnerVolumeSpecName "kube-api-access-4dz55". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:34.615622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.600523 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa53b017-4e9c-4da6-944a-d880e98f3cda-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.643621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.637175 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dz55\" (UniqueName: \"kubernetes.io/projected/aa53b017-4e9c-4da6-944a-d880e98f3cda-kube-api-access-4dz55\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:34.732673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.731787 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7"] Oct 31 13:31:34.796627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.776689 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:31:34.891797 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.891751 5532 patch_prober.go:29] interesting pod/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 13:31:34.892040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.892013 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 13:31:34.915229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:34.915021 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8439b78a-76cd9748d5-g26f7"] Oct 31 13:31:35.323167 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:35.322848 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" Oct 31 13:31:35.326467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:35.325156 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj" event={"ID":"aa53b017-4e9c-4da6-944a-d880e98f3cda","Type":"ContainerDied","Data":"3aa3c49bf84d51f25eee799d4eef92d0cea64b3cb92ebdf8ece821299cde8ed1"} Oct 31 13:31:35.326467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:35.325261 5532 scope.go:122] "RemoveContainer" containerID="4e2b95c488d317dbad573b3b7abf9984c2a7435763cb271367fb4fbf60bc4c73" Oct 31 13:31:35.338532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:35.338502 5532 patch_prober.go:29] interesting pod/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:31:35.338685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:35.338555 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 13:31:35.399302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:35.399269 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj"] Oct 31 13:31:35.404440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:35.404414 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-fc1bbe2b-fc58f6595-7wwxj"] Oct 31 13:31:35.770611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:35.770557 5532 patch_prober.go:29] interesting pod/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 13:31:35.788402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:35.788361 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 13:31:36.006529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:36.006496 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373a9ad9-0eac-4420-876b-c0ad17235652" path="/var/lib/kubelet/pods/373a9ad9-0eac-4420-876b-c0ad17235652/volumes" Oct 31 13:31:36.007129 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:36.007110 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa53b017-4e9c-4da6-944a-d880e98f3cda" path="/var/lib/kubelet/pods/aa53b017-4e9c-4da6-944a-d880e98f3cda/volumes" Oct 31 13:31:36.337194 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:36.337154 5532 patch_prober.go:29] interesting pod/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:31:36.337732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:36.337212 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 13:31:36.516096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:36.515965 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2"] Oct 31 13:31:36.516322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:36.516290 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" podUID="2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5" containerName="maas-api" containerID="cri-o://a82c7b81afc401a2f2b966c9a3efa7d32aa0233d123ab33d55dd026ed0bc5c44" gracePeriod=30 Oct 31 13:31:36.784617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:36.784502 5532 patch_prober.go:29] interesting pod/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 13:31:36.784617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:36.784567 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 13:31:37.182620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.174022 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd"] Oct 31 13:31:37.549711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.534692 5532 patch_prober.go:29] interesting pod/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:31:37.549711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.534797 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 13:31:37.633574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.633537 5532 generic.go:356] "Generic (PLEG): container finished" podID="2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5" containerID="a82c7b81afc401a2f2b966c9a3efa7d32aa0233d123ab33d55dd026ed0bc5c44" exitCode=0 Oct 31 13:31:37.633907 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.633880 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" event={"ID":"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5","Type":"ContainerDied","Data":"a82c7b81afc401a2f2b966c9a3efa7d32aa0233d123ab33d55dd026ed0bc5c44"} Oct 31 13:31:37.756035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.738028 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-fc1bbe2b-r2rbd"] Oct 31 13:31:37.835619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.801762 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z"] Oct 31 13:31:37.835619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.803828 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c56f93a4-4c00-4b39-b175-952105ec043d" containerName="revoke-keys" Oct 31 13:31:37.835619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.803851 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c56f93a4-4c00-4b39-b175-952105ec043d" containerName="revoke-keys" Oct 31 13:31:37.835619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.803885 5532 cpu_manager.go:414] "containerMap: removing container" podUID="aa53b017-4e9c-4da6-944a-d880e98f3cda" containerName="payload-pre-processing" Oct 31 13:31:37.835619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.803893 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa53b017-4e9c-4da6-944a-d880e98f3cda" containerName="payload-pre-processing" Oct 31 13:31:37.835619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.803918 5532 cpu_manager.go:414] "containerMap: removing container" podUID="373a9ad9-0eac-4420-876b-c0ad17235652" containerName="maas-api" Oct 31 13:31:37.835619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.803926 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373a9ad9-0eac-4420-876b-c0ad17235652" containerName="maas-api" Oct 31 13:31:37.835619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.804495 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c56f93a4-4c00-4b39-b175-952105ec043d" containerName="revoke-keys" Oct 31 13:31:37.835619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.804531 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa53b017-4e9c-4da6-944a-d880e98f3cda" containerName="payload-pre-processing" Oct 31 13:31:37.835619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.804543 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="373a9ad9-0eac-4420-876b-c0ad17235652" containerName="maas-api" Oct 31 13:31:37.877771 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.877728 5532 patch_prober.go:29] interesting pod/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 13:31:37.967237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:37.966714 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:38.008744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.008681 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 13:31:38.084917 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.084823 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" Oct 31 13:31:38.105664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.101077 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:31:38.117359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.112693 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:31:38.256469 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.231739 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:38.269933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.238324 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c56f93a4-4c00-4b39-b175-952105ec043d" path="/var/lib/kubelet/pods/c56f93a4-4c00-4b39-b175-952105ec043d/volumes" Oct 31 13:31:38.359278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.346341 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-kube-api-access-84z4m\" (UniqueName: \"kubernetes.io/projected/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-kube-api-access-84z4m\") pod \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " Oct 31 13:31:38.359278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.346498 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-metrics-certs\") pod \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " Oct 31 13:31:38.359278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.346543 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-maas-api-tls\") pod \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " Oct 31 13:31:38.388621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.385660 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-openshift-service-ca\") pod \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\" (UID: \"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5\") " Oct 31 13:31:38.389030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.388993 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-openshift-service-ca" pod "2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5" (UID: "2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:38.433620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.410423 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-maas-api-tls" pod "2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5" (UID: "2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:38.433620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.412025 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcjzr\" (UniqueName: \"kubernetes.io/projected/685d6144-a686-4134-98a9-154bede4333b-kube-api-access-bcjzr\") pod \"maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z\" (UID: \"685d6144-a686-4134-98a9-154bede4333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" Oct 31 13:31:38.433620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.412141 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685d6144-a686-4134-98a9-154bede4333b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z\" (UID: \"685d6144-a686-4134-98a9-154bede4333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" Oct 31 13:31:38.433620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.412457 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685d6144-a686-4134-98a9-154bede4333b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z\" (UID: \"685d6144-a686-4134-98a9-154bede4333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" Oct 31 13:31:38.433620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.412645 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:38.433620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.412665 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:38.474697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.460705 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 13:31:38.495076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.495037 5532 patch_prober.go:29] interesting pod/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" start-of-body= Oct 31 13:31:38.511169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.511127 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.113:15021/healthz/ready\": dial tcp 10.132.0.113:15021: connect: connection refused" Oct 31 13:31:38.512503 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.512472 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z"] Oct 31 13:31:38.541960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.541929 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r"] Oct 31 13:31:38.542479 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.542436 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" containerID="cri-o://d27b17c4da64a6fb4758670e3fa2df1d9240dfa7759aa15ea4ccb9fc10525bb8" gracePeriod=30 Oct 31 13:31:38.542785 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.537002 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcjzr\" (UniqueName: \"kubernetes.io/projected/685d6144-a686-4134-98a9-154bede4333b-kube-api-access-bcjzr\") pod \"maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z\" (UID: \"685d6144-a686-4134-98a9-154bede4333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" Oct 31 13:31:38.543101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.543076 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685d6144-a686-4134-98a9-154bede4333b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z\" (UID: \"685d6144-a686-4134-98a9-154bede4333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" Oct 31 13:31:38.543557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.543534 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685d6144-a686-4134-98a9-154bede4333b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z\" (UID: \"685d6144-a686-4134-98a9-154bede4333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" Oct 31 13:31:38.553688 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.544649 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685d6144-a686-4134-98a9-154bede4333b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z\" (UID: \"685d6144-a686-4134-98a9-154bede4333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" Oct 31 13:31:38.745021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.714385 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-metrics-certs" pod "2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5" (UID: "2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:38.745021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.714662 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-kube-api-access-84z4m" pod "2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5" (UID: "2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5"). InnerVolumeSpecName "kube-api-access-84z4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:38.746940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.746898 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcjzr\" (UniqueName: \"kubernetes.io/projected/685d6144-a686-4134-98a9-154bede4333b-kube-api-access-bcjzr\") pod \"maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z\" (UID: \"685d6144-a686-4134-98a9-154bede4333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" Oct 31 13:31:38.793719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.769024 5532 patch_prober.go:29] interesting pod/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 31 13:31:38.793719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.769085 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 31 13:31:38.811431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.811396 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685d6144-a686-4134-98a9-154bede4333b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z\" (UID: \"685d6144-a686-4134-98a9-154bede4333b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" Oct 31 13:31:38.847125 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.831958 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84z4m\" (UniqueName: \"kubernetes.io/projected/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-kube-api-access-84z4m\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:38.847125 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.831995 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:38.847125 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.832684 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" Oct 31 13:31:38.872076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.872028 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" event={"ID":"2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5","Type":"ContainerDied","Data":"be6b59c37531dc5671daa3f29a3f879a612b9c8b8cfc97062c723d13135071cf"} Oct 31 13:31:38.880530 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.880501 5532 scope.go:122] "RemoveContainer" containerID="a82c7b81afc401a2f2b966c9a3efa7d32aa0233d123ab33d55dd026ed0bc5c44" Oct 31 13:31:38.892620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:38.890804 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2" Oct 31 13:31:39.053665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.050976 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4"] Oct 31 13:31:39.053665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.051223 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" containerName="istio-proxy" containerID="cri-o://53bfa6de10ea0312ccab1eed08465909bb25e18e61442c6bc856c69359ac35bc" gracePeriod=30 Oct 31 13:31:39.193280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.192746 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:39.530233 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.529229 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:31:39.555573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.554710 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:39.614690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.614648 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z"] Oct 31 13:31:39.630668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.616131 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5" containerName="maas-api" Oct 31 13:31:39.634026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.633982 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5" containerName="maas-api" Oct 31 13:31:39.634568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.634522 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5" containerName="maas-api" Oct 31 13:31:39.677939 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.677906 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:39.688799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.688767 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z"] Oct 31 13:31:39.720781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.714232 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-4f340c-openshift-default-dockercfg-2nqnk\"" Oct 31 13:31:39.731852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.731741 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2"] Oct 31 13:31:39.786617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.784527 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:39.866329 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.866270 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:39.896616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.868852 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dd044e-3702-46d9-91db-2d456c9dd345-istiod-ca-cert\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:39.896616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.868970 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-envoy\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:39.896616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.869006 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-token\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:39.896616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.869094 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-certs\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:39.896616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.869187 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b7tl\" (UniqueName: \"kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-kube-api-access-8b7tl\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:39.896616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.869250 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-socket\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:39.896616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.869327 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-data\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:39.896616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.869494 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-credential-socket\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:39.896616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:39.869524 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-podinfo\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.040929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.040838 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:31:40.080687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.069868 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b7tl\" (UniqueName: \"kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-kube-api-access-8b7tl\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.080687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.069953 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-socket\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.080687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.070030 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-data\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.080687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.070180 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-credential-socket\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.080687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.070214 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-podinfo\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.080687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.070282 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dd044e-3702-46d9-91db-2d456c9dd345-istiod-ca-cert\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.080687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.070381 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-envoy\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.080687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.070410 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-token\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.080687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.070496 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-certs\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.080687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.071007 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-certs\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.109976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.109943 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-data\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.110633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.110586 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-credential-socket\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.124751 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.124707 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-socket\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.185617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.181928 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:31:40.203422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.202154 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:31:40.251103 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.204778 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:40.251103 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.244710 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-envoy\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.285370 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.279484 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-fc1bbe2b-76568f6b7-wqvs2"] Oct 31 13:31:40.285370 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.279584 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:31:40.292854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.292782 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dd044e-3702-46d9-91db-2d456c9dd345-istiod-ca-cert\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.318507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.296274 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-token\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.342677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.342543 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:40.460105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.460018 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z"] Oct 31 13:31:40.531477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.529333 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-podinfo\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.531477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.529436 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:40.531477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.529935 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:31:40.619658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.556516 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:40.650185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.647449 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:31:40.687622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.676799 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b7tl\" (UniqueName: \"kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-kube-api-access-8b7tl\") pod \"e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.710622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.688427 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:40.916140 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.907906 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:31:40.968360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:40.968317 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:31:41.002452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:41.002418 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:31:41.006753 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:41.006718 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" event={"ID":"685d6144-a686-4134-98a9-154bede4333b","Type":"ContainerStarted","Data":"ea9e5b8a7d6cf819afd0022e6f15fdec766612aa0cd862933d43f05fc7d3e76d"} Oct 31 13:31:41.087230 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:41.083402 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:31:41.281971 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:31:41.281939 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3dd044e_3702_46d9_91db_2d456c9dd345.slice/crio-1a56626c2589590e0a0f86eeddb40c695fbc565d211db208a1b7b90ec9f44a5d WatchSource:0}: Error finding container 1a56626c2589590e0a0f86eeddb40c695fbc565d211db208a1b7b90ec9f44a5d: Status 404 returned error can't find the container with id 1a56626c2589590e0a0f86eeddb40c695fbc565d211db208a1b7b90ec9f44a5d Oct 31 13:31:41.283414 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:41.283391 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z"] Oct 31 13:31:41.284873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:41.284839 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:31:41.284952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:41.284909 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:31:41.284952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:41.284939 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:31:42.006388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:42.006354 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5" path="/var/lib/kubelet/pods/2dd4fba5-acda-4071-bc7d-f9b4ae0a4ca5/volumes" Oct 31 13:31:42.016894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:42.016865 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" event={"ID":"685d6144-a686-4134-98a9-154bede4333b","Type":"ContainerStarted","Data":"5d8a4afda6275c49a05d4651ad61e4f32bb13ca7362ee4b70d3312a58df2fe82"} Oct 31 13:31:42.018668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:42.018642 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" event={"ID":"a3dd044e-3702-46d9-91db-2d456c9dd345","Type":"ContainerStarted","Data":"aac1668030455d43982c62c2232c93ad7105a4b718e8b8430cd35d2636767b9b"} Oct 31 13:31:42.018792 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:42.018690 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" event={"ID":"a3dd044e-3702-46d9-91db-2d456c9dd345","Type":"ContainerStarted","Data":"1a56626c2589590e0a0f86eeddb40c695fbc565d211db208a1b7b90ec9f44a5d"} Oct 31 13:31:42.050779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:42.050723 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" podStartSLOduration=5.050708118 podStartE2EDuration="5.050708118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:42.050168805 +0000 UTC m=+1670.195912638" watchObservedRunningTime="2026-10-31 13:31:42.050708118 +0000 UTC m=+1670.196451945" Oct 31 13:31:42.085702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:42.085632 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" podStartSLOduration=3.085613443 podStartE2EDuration="3.085613443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:42.082287472 +0000 UTC m=+1670.228031300" watchObservedRunningTime="2026-10-31 13:31:42.085613443 +0000 UTC m=+1670.231357263" Oct 31 13:31:42.133391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:42.133320 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq"] Oct 31 13:31:42.140115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:42.140052 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52c82151-62hpq"] Oct 31 13:31:42.701934 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:42.701904 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:42.707497 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:42.707472 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:43.051739 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:43.051648 5532 generic.go:356] "Generic (PLEG): container finished" podID="685d6144-a686-4134-98a9-154bede4333b" containerID="5d8a4afda6275c49a05d4651ad61e4f32bb13ca7362ee4b70d3312a58df2fe82" exitCode=0 Oct 31 13:31:43.052262 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:43.051765 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" event={"ID":"685d6144-a686-4134-98a9-154bede4333b","Type":"ContainerDied","Data":"5d8a4afda6275c49a05d4651ad61e4f32bb13ca7362ee4b70d3312a58df2fe82"} Oct 31 13:31:43.052262 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:43.052235 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:43.053735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:43.053711 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:31:43.934075 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:43.934043 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:31:44.046316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:44.045785 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99e9d6a-eeec-490b-8d43-86199491b259" path="/var/lib/kubelet/pods/a99e9d6a-eeec-490b-8d43-86199491b259/volumes" Oct 31 13:31:44.253365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:44.253342 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" Oct 31 13:31:44.308512 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:44.308475 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/685d6144-a686-4134-98a9-154bede4333b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685d6144-a686-4134-98a9-154bede4333b-maas-api-xks-ca\") pod \"685d6144-a686-4134-98a9-154bede4333b\" (UID: \"685d6144-a686-4134-98a9-154bede4333b\") " Oct 31 13:31:44.308804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:44.308780 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/685d6144-a686-4134-98a9-154bede4333b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685d6144-a686-4134-98a9-154bede4333b-maas-api-openshift-ca\") pod \"685d6144-a686-4134-98a9-154bede4333b\" (UID: \"685d6144-a686-4134-98a9-154bede4333b\") " Oct 31 13:31:44.308912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:44.308891 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/685d6144-a686-4134-98a9-154bede4333b-kube-api-access-bcjzr\" (UniqueName: \"kubernetes.io/projected/685d6144-a686-4134-98a9-154bede4333b-kube-api-access-bcjzr\") pod \"685d6144-a686-4134-98a9-154bede4333b\" (UID: \"685d6144-a686-4134-98a9-154bede4333b\") " Oct 31 13:31:44.309193 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:44.309162 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/685d6144-a686-4134-98a9-154bede4333b-maas-api-openshift-ca" pod "685d6144-a686-4134-98a9-154bede4333b" (UID: "685d6144-a686-4134-98a9-154bede4333b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:44.309532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:44.309513 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/685d6144-a686-4134-98a9-154bede4333b-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:44.311044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:44.311012 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/685d6144-a686-4134-98a9-154bede4333b-maas-api-xks-ca" pod "685d6144-a686-4134-98a9-154bede4333b" (UID: "685d6144-a686-4134-98a9-154bede4333b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:31:44.311159 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:44.311076 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685d6144-a686-4134-98a9-154bede4333b-kube-api-access-bcjzr" pod "685d6144-a686-4134-98a9-154bede4333b" (UID: "685d6144-a686-4134-98a9-154bede4333b"). InnerVolumeSpecName "kube-api-access-bcjzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:44.411583 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:44.411550 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcjzr\" (UniqueName: \"kubernetes.io/projected/685d6144-a686-4134-98a9-154bede4333b-kube-api-access-bcjzr\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:44.411583 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:44.411580 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/685d6144-a686-4134-98a9-154bede4333b-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:45.072867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.072829 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" Oct 31 13:31:45.072867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.072853 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z" event={"ID":"685d6144-a686-4134-98a9-154bede4333b","Type":"ContainerDied","Data":"ea9e5b8a7d6cf819afd0022e6f15fdec766612aa0cd862933d43f05fc7d3e76d"} Oct 31 13:31:45.073066 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.072903 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea9e5b8a7d6cf819afd0022e6f15fdec766612aa0cd862933d43f05fc7d3e76d" Oct 31 13:31:45.823134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.823096 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4"] Oct 31 13:31:45.823910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.823894 5532 cpu_manager.go:414] "containerMap: removing container" podUID="685d6144-a686-4134-98a9-154bede4333b" containerName="revoke-keys" Oct 31 13:31:45.823910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.823911 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="685d6144-a686-4134-98a9-154bede4333b" containerName="revoke-keys" Oct 31 13:31:45.824042 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.824032 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="685d6144-a686-4134-98a9-154bede4333b" containerName="revoke-keys" Oct 31 13:31:45.828217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.828195 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" Oct 31 13:31:45.836626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.836577 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-4f340c\"" Oct 31 13:31:45.842960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.842936 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4"] Oct 31 13:31:45.845501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.845481 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-4f340c-dockercfg-8sfgc\"" Oct 31 13:31:45.892209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.892158 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d"] Oct 31 13:31:45.897563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.897537 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" Oct 31 13:31:45.911952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.911577 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d"] Oct 31 13:31:45.937466 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.937434 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4\" (UID: \"2cce49b5-22bd-4655-90eb-75b5fb6cae5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" Oct 31 13:31:45.937639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.937528 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfbk2\" (UniqueName: \"kubernetes.io/projected/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-kube-api-access-qfbk2\") pod \"payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4\" (UID: \"2cce49b5-22bd-4655-90eb-75b5fb6cae5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" Oct 31 13:31:45.937701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.937690 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz9nl\" (UniqueName: \"kubernetes.io/projected/01c8f8ae-be64-4501-adb5-dc2689f938bf-kube-api-access-pz9nl\") pod \"payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d\" (UID: \"01c8f8ae-be64-4501-adb5-dc2689f938bf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" Oct 31 13:31:45.937744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:45.937732 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c8f8ae-be64-4501-adb5-dc2689f938bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d\" (UID: \"01c8f8ae-be64-4501-adb5-dc2689f938bf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" Oct 31 13:31:46.039676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.039576 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4\" (UID: \"2cce49b5-22bd-4655-90eb-75b5fb6cae5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" Oct 31 13:31:46.039869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.039724 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfbk2\" (UniqueName: \"kubernetes.io/projected/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-kube-api-access-qfbk2\") pod \"payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4\" (UID: \"2cce49b5-22bd-4655-90eb-75b5fb6cae5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" Oct 31 13:31:46.039939 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.039869 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz9nl\" (UniqueName: \"kubernetes.io/projected/01c8f8ae-be64-4501-adb5-dc2689f938bf-kube-api-access-pz9nl\") pod \"payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d\" (UID: \"01c8f8ae-be64-4501-adb5-dc2689f938bf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" Oct 31 13:31:46.039999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.039934 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c8f8ae-be64-4501-adb5-dc2689f938bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d\" (UID: \"01c8f8ae-be64-4501-adb5-dc2689f938bf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" Oct 31 13:31:46.058956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.041007 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c8f8ae-be64-4501-adb5-dc2689f938bf-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d\" (UID: \"01c8f8ae-be64-4501-adb5-dc2689f938bf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" Oct 31 13:31:46.058956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.041755 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4\" (UID: \"2cce49b5-22bd-4655-90eb-75b5fb6cae5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" Oct 31 13:31:46.119665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.119420 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz9nl\" (UniqueName: \"kubernetes.io/projected/01c8f8ae-be64-4501-adb5-dc2689f938bf-kube-api-access-pz9nl\") pod \"payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d\" (UID: \"01c8f8ae-be64-4501-adb5-dc2689f938bf\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" Oct 31 13:31:46.136002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.126140 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfbk2\" (UniqueName: \"kubernetes.io/projected/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-kube-api-access-qfbk2\") pod \"payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4\" (UID: \"2cce49b5-22bd-4655-90eb-75b5fb6cae5b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" Oct 31 13:31:46.172617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.155913 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" Oct 31 13:31:46.218333 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.217995 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" Oct 31 13:31:46.530433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.530387 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:31:46.580957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.580927 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4"] Oct 31 13:31:46.582614 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:31:46.581101 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cce49b5_22bd_4655_90eb_75b5fb6cae5b.slice/crio-85a029ec973da22f667477e40d794e3e58a545b2491cda14fbfbd929bfc90a4c WatchSource:0}: Error finding container 85a029ec973da22f667477e40d794e3e58a545b2491cda14fbfbd929bfc90a4c: Status 404 returned error can't find the container with id 85a029ec973da22f667477e40d794e3e58a545b2491cda14fbfbd929bfc90a4c Oct 31 13:31:46.588636 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:31:46.588581 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c8f8ae_be64_4501_adb5_dc2689f938bf.slice/crio-f517e42202a9501751db3d304f11d508a92530aab071a7be676d475e38bfc541 WatchSource:0}: Error finding container f517e42202a9501751db3d304f11d508a92530aab071a7be676d475e38bfc541: Status 404 returned error can't find the container with id f517e42202a9501751db3d304f11d508a92530aab071a7be676d475e38bfc541 Oct 31 13:31:46.589919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.589889 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d"] Oct 31 13:31:46.654335 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.654307 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/255e3805-e523-4620-b06a-f4d8fa91538f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255e3805-e523-4620-b06a-f4d8fa91538f-istiod-ca-cert\") pod \"255e3805-e523-4620-b06a-f4d8fa91538f\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " Oct 31 13:31:46.654462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.654355 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-credential-socket\") pod \"255e3805-e523-4620-b06a-f4d8fa91538f\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " Oct 31 13:31:46.654462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.654421 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-envoy\") pod \"255e3805-e523-4620-b06a-f4d8fa91538f\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " Oct 31 13:31:46.654606 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.654465 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-socket\") pod \"255e3805-e523-4620-b06a-f4d8fa91538f\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " Oct 31 13:31:46.654606 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.654508 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-data\") pod \"255e3805-e523-4620-b06a-f4d8fa91538f\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " Oct 31 13:31:46.654716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.654584 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/255e3805-e523-4620-b06a-f4d8fa91538f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255e3805-e523-4620-b06a-f4d8fa91538f-istio-podinfo\") pod \"255e3805-e523-4620-b06a-f4d8fa91538f\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " Oct 31 13:31:46.654767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.654725 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-certs\") pod \"255e3805-e523-4620-b06a-f4d8fa91538f\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " Oct 31 13:31:46.654767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.654725 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-credential-socket" pod "255e3805-e523-4620-b06a-f4d8fa91538f" (UID: "255e3805-e523-4620-b06a-f4d8fa91538f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:46.654767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.654743 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/255e3805-e523-4620-b06a-f4d8fa91538f-istiod-ca-cert" pod "255e3805-e523-4620-b06a-f4d8fa91538f" (UID: "255e3805-e523-4620-b06a-f4d8fa91538f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:46.654918 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.654774 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-kube-api-access-t876h\" (UniqueName: \"kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-kube-api-access-t876h\") pod \"255e3805-e523-4620-b06a-f4d8fa91538f\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " Oct 31 13:31:46.654970 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.654918 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-socket" pod "255e3805-e523-4620-b06a-f4d8fa91538f" (UID: "255e3805-e523-4620-b06a-f4d8fa91538f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:46.655021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.655011 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-istio-token\" (UniqueName: \"kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-istio-token\") pod \"255e3805-e523-4620-b06a-f4d8fa91538f\" (UID: \"255e3805-e523-4620-b06a-f4d8fa91538f\") " Oct 31 13:31:46.655080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.655022 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-certs" pod "255e3805-e523-4620-b06a-f4d8fa91538f" (UID: "255e3805-e523-4620-b06a-f4d8fa91538f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:46.655080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.655057 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-data" pod "255e3805-e523-4620-b06a-f4d8fa91538f" (UID: "255e3805-e523-4620-b06a-f4d8fa91538f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:46.655648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.655629 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/255e3805-e523-4620-b06a-f4d8fa91538f-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.655750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.655652 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.655750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.655666 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.655750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.655679 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.655750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.655693 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.657275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.657251 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-envoy" pod "255e3805-e523-4620-b06a-f4d8fa91538f" (UID: "255e3805-e523-4620-b06a-f4d8fa91538f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:46.657359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.657342 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/255e3805-e523-4620-b06a-f4d8fa91538f-istio-podinfo" pod "255e3805-e523-4620-b06a-f4d8fa91538f" (UID: "255e3805-e523-4620-b06a-f4d8fa91538f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:31:46.657416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.657357 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-kube-api-access-t876h" pod "255e3805-e523-4620-b06a-f4d8fa91538f" (UID: "255e3805-e523-4620-b06a-f4d8fa91538f"). InnerVolumeSpecName "kube-api-access-t876h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:46.658476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.658456 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-istio-token" pod "255e3805-e523-4620-b06a-f4d8fa91538f" (UID: "255e3805-e523-4620-b06a-f4d8fa91538f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:46.757171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.757132 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.757171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.757160 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/255e3805-e523-4620-b06a-f4d8fa91538f-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.757171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.757170 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/255e3805-e523-4620-b06a-f4d8fa91538f-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.757171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.757179 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t876h\" (UniqueName: \"kubernetes.io/projected/255e3805-e523-4620-b06a-f4d8fa91538f-kube-api-access-t876h\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:46.976265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:46.976236 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:31:47.064849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.064728 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-data\") pod \"204317d2-4400-4923-adac-e410dced28b3\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " Oct 31 13:31:47.064849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.064836 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-credential-socket\") pod \"204317d2-4400-4923-adac-e410dced28b3\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " Oct 31 13:31:47.065078 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.064861 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/204317d2-4400-4923-adac-e410dced28b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204317d2-4400-4923-adac-e410dced28b3-istiod-ca-cert\") pod \"204317d2-4400-4923-adac-e410dced28b3\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " Oct 31 13:31:47.065121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.065086 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-data" pod "204317d2-4400-4923-adac-e410dced28b3" (UID: "204317d2-4400-4923-adac-e410dced28b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:47.065121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.065108 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-envoy\") pod \"204317d2-4400-4923-adac-e410dced28b3\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " Oct 31 13:31:47.065218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.065145 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-kube-api-access-nfc5n\" (UniqueName: \"kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-kube-api-access-nfc5n\") pod \"204317d2-4400-4923-adac-e410dced28b3\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " Oct 31 13:31:47.065218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.065127 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-credential-socket" pod "204317d2-4400-4923-adac-e410dced28b3" (UID: "204317d2-4400-4923-adac-e410dced28b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:47.065331 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.065238 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-certs\") pod \"204317d2-4400-4923-adac-e410dced28b3\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " Oct 31 13:31:47.065331 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.065274 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/204317d2-4400-4923-adac-e410dced28b3-istiod-ca-cert" pod "204317d2-4400-4923-adac-e410dced28b3" (UID: "204317d2-4400-4923-adac-e410dced28b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:47.065331 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.065297 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-socket\") pod \"204317d2-4400-4923-adac-e410dced28b3\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " Oct 31 13:31:47.065473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.065335 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/204317d2-4400-4923-adac-e410dced28b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204317d2-4400-4923-adac-e410dced28b3-istio-podinfo\") pod \"204317d2-4400-4923-adac-e410dced28b3\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " Oct 31 13:31:47.065473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.065380 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-istio-token\" (UniqueName: \"kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-istio-token\") pod \"204317d2-4400-4923-adac-e410dced28b3\" (UID: \"204317d2-4400-4923-adac-e410dced28b3\") " Oct 31 13:31:47.065649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.065532 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-certs" pod "204317d2-4400-4923-adac-e410dced28b3" (UID: "204317d2-4400-4923-adac-e410dced28b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:47.065649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.065560 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-socket" pod "204317d2-4400-4923-adac-e410dced28b3" (UID: "204317d2-4400-4923-adac-e410dced28b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:47.066116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.066094 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:47.066116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.066115 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:47.066283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.066125 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:47.066283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.066134 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:47.066283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.066143 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/204317d2-4400-4923-adac-e410dced28b3-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:47.067467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.067443 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-envoy" pod "204317d2-4400-4923-adac-e410dced28b3" (UID: "204317d2-4400-4923-adac-e410dced28b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:31:47.067553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.067459 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-istio-token" pod "204317d2-4400-4923-adac-e410dced28b3" (UID: "204317d2-4400-4923-adac-e410dced28b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:47.067710 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.067693 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-kube-api-access-nfc5n" pod "204317d2-4400-4923-adac-e410dced28b3" (UID: "204317d2-4400-4923-adac-e410dced28b3"). InnerVolumeSpecName "kube-api-access-nfc5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:47.067755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.067706 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/204317d2-4400-4923-adac-e410dced28b3-istio-podinfo" pod "204317d2-4400-4923-adac-e410dced28b3" (UID: "204317d2-4400-4923-adac-e410dced28b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:31:47.128004 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.127964 5532 generic.go:356] "Generic (PLEG): container finished" podID="204317d2-4400-4923-adac-e410dced28b3" containerID="d27b17c4da64a6fb4758670e3fa2df1d9240dfa7759aa15ea4ccb9fc10525bb8" exitCode=0 Oct 31 13:31:47.128300 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.128068 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" event={"ID":"204317d2-4400-4923-adac-e410dced28b3","Type":"ContainerDied","Data":"d27b17c4da64a6fb4758670e3fa2df1d9240dfa7759aa15ea4ccb9fc10525bb8"} Oct 31 13:31:47.128300 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.128149 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" event={"ID":"204317d2-4400-4923-adac-e410dced28b3","Type":"ContainerDied","Data":"8420821e84f5c916d1230b39d71b5c1ebcbc0009c0daf03375e06f17abc635ae"} Oct 31 13:31:47.128300 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.128179 5532 scope.go:122] "RemoveContainer" containerID="d27b17c4da64a6fb4758670e3fa2df1d9240dfa7759aa15ea4ccb9fc10525bb8" Oct 31 13:31:47.128478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.128320 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r" Oct 31 13:31:47.131347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.131312 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" event={"ID":"01c8f8ae-be64-4501-adb5-dc2689f938bf","Type":"ContainerStarted","Data":"7b31504b1944e6e8a0f48b52c5a94a88a733e7f43ce514929fc6bd5d00dc9f63"} Oct 31 13:31:47.131486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.131362 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" event={"ID":"01c8f8ae-be64-4501-adb5-dc2689f938bf","Type":"ContainerStarted","Data":"f517e42202a9501751db3d304f11d508a92530aab071a7be676d475e38bfc541"} Oct 31 13:31:47.132244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.132207 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" Oct 31 13:31:47.135437 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.135412 5532 generic.go:356] "Generic (PLEG): container finished" podID="255e3805-e523-4620-b06a-f4d8fa91538f" containerID="53bfa6de10ea0312ccab1eed08465909bb25e18e61442c6bc856c69359ac35bc" exitCode=0 Oct 31 13:31:47.135533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.135462 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" Oct 31 13:31:47.135533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.135516 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" event={"ID":"255e3805-e523-4620-b06a-f4d8fa91538f","Type":"ContainerDied","Data":"53bfa6de10ea0312ccab1eed08465909bb25e18e61442c6bc856c69359ac35bc"} Oct 31 13:31:47.135746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.135695 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4" event={"ID":"255e3805-e523-4620-b06a-f4d8fa91538f","Type":"ContainerDied","Data":"1a9d5cb30f9a829d9175eb9fc011d0270b1a8be9404199e01e712c4162958064"} Oct 31 13:31:47.140052 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.140025 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" event={"ID":"2cce49b5-22bd-4655-90eb-75b5fb6cae5b","Type":"ContainerStarted","Data":"c3a59b3d1bf64ecda52acdb1eb1280f9995472a7f35d5f731b3461a09df3fcc0"} Oct 31 13:31:47.140159 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.140072 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" event={"ID":"2cce49b5-22bd-4655-90eb-75b5fb6cae5b","Type":"ContainerStarted","Data":"85a029ec973da22f667477e40d794e3e58a545b2491cda14fbfbd929bfc90a4c"} Oct 31 13:31:47.140810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.140562 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" Oct 31 13:31:47.162310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.162292 5532 scope.go:122] "RemoveContainer" containerID="5b708277ce190efd946845937b0874887e185b0b25590dbf88b72c65c91d96c7" Oct 31 13:31:47.163504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.163448 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" podStartSLOduration=2.163432691 podStartE2EDuration="2.163432691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:47.157706001 +0000 UTC m=+1675.303449821" watchObservedRunningTime="2026-10-31 13:31:47.163432691 +0000 UTC m=+1675.309176520" Oct 31 13:31:47.168963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.168944 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/204317d2-4400-4923-adac-e410dced28b3-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:47.169036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.168965 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfc5n\" (UniqueName: \"kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-kube-api-access-nfc5n\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:47.169036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.168975 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/204317d2-4400-4923-adac-e410dced28b3-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:47.169036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.168984 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/204317d2-4400-4923-adac-e410dced28b3-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:47.173474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.173456 5532 scope.go:122] "RemoveContainer" containerID="d27b17c4da64a6fb4758670e3fa2df1d9240dfa7759aa15ea4ccb9fc10525bb8" Oct 31 13:31:47.173780 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:31:47.173757 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27b17c4da64a6fb4758670e3fa2df1d9240dfa7759aa15ea4ccb9fc10525bb8\": container with ID starting with d27b17c4da64a6fb4758670e3fa2df1d9240dfa7759aa15ea4ccb9fc10525bb8 not found: ID does not exist" containerID="d27b17c4da64a6fb4758670e3fa2df1d9240dfa7759aa15ea4ccb9fc10525bb8" Oct 31 13:31:47.173866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.173785 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27b17c4da64a6fb4758670e3fa2df1d9240dfa7759aa15ea4ccb9fc10525bb8"} err="failed to get container status \"d27b17c4da64a6fb4758670e3fa2df1d9240dfa7759aa15ea4ccb9fc10525bb8\": rpc error: code = NotFound desc = could not find container \"d27b17c4da64a6fb4758670e3fa2df1d9240dfa7759aa15ea4ccb9fc10525bb8\": container with ID starting with d27b17c4da64a6fb4758670e3fa2df1d9240dfa7759aa15ea4ccb9fc10525bb8 not found: ID does not exist" Oct 31 13:31:47.173866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.173802 5532 scope.go:122] "RemoveContainer" containerID="5b708277ce190efd946845937b0874887e185b0b25590dbf88b72c65c91d96c7" Oct 31 13:31:47.174035 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:31:47.174019 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b708277ce190efd946845937b0874887e185b0b25590dbf88b72c65c91d96c7\": container with ID starting with 5b708277ce190efd946845937b0874887e185b0b25590dbf88b72c65c91d96c7 not found: ID does not exist" containerID="5b708277ce190efd946845937b0874887e185b0b25590dbf88b72c65c91d96c7" Oct 31 13:31:47.174082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.174036 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b708277ce190efd946845937b0874887e185b0b25590dbf88b72c65c91d96c7"} err="failed to get container status \"5b708277ce190efd946845937b0874887e185b0b25590dbf88b72c65c91d96c7\": rpc error: code = NotFound desc = could not find container \"5b708277ce190efd946845937b0874887e185b0b25590dbf88b72c65c91d96c7\": container with ID starting with 5b708277ce190efd946845937b0874887e185b0b25590dbf88b72c65c91d96c7 not found: ID does not exist" Oct 31 13:31:47.174082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.174048 5532 scope.go:122] "RemoveContainer" containerID="53bfa6de10ea0312ccab1eed08465909bb25e18e61442c6bc856c69359ac35bc" Oct 31 13:31:47.184312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.184153 5532 scope.go:122] "RemoveContainer" containerID="b5c85baeaf8e3bc4d53903705543dfe2296d252cd396456529c5676604641255" Oct 31 13:31:47.194193 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.194174 5532 scope.go:122] "RemoveContainer" containerID="53bfa6de10ea0312ccab1eed08465909bb25e18e61442c6bc856c69359ac35bc" Oct 31 13:31:47.194494 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:31:47.194469 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53bfa6de10ea0312ccab1eed08465909bb25e18e61442c6bc856c69359ac35bc\": container with ID starting with 53bfa6de10ea0312ccab1eed08465909bb25e18e61442c6bc856c69359ac35bc not found: ID does not exist" containerID="53bfa6de10ea0312ccab1eed08465909bb25e18e61442c6bc856c69359ac35bc" Oct 31 13:31:47.194569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.194504 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53bfa6de10ea0312ccab1eed08465909bb25e18e61442c6bc856c69359ac35bc"} err="failed to get container status \"53bfa6de10ea0312ccab1eed08465909bb25e18e61442c6bc856c69359ac35bc\": rpc error: code = NotFound desc = could not find container \"53bfa6de10ea0312ccab1eed08465909bb25e18e61442c6bc856c69359ac35bc\": container with ID starting with 53bfa6de10ea0312ccab1eed08465909bb25e18e61442c6bc856c69359ac35bc not found: ID does not exist" Oct 31 13:31:47.194569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.194528 5532 scope.go:122] "RemoveContainer" containerID="b5c85baeaf8e3bc4d53903705543dfe2296d252cd396456529c5676604641255" Oct 31 13:31:47.194890 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:31:47.194843 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5c85baeaf8e3bc4d53903705543dfe2296d252cd396456529c5676604641255\": container with ID starting with b5c85baeaf8e3bc4d53903705543dfe2296d252cd396456529c5676604641255 not found: ID does not exist" containerID="b5c85baeaf8e3bc4d53903705543dfe2296d252cd396456529c5676604641255" Oct 31 13:31:47.194890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.194871 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5c85baeaf8e3bc4d53903705543dfe2296d252cd396456529c5676604641255"} err="failed to get container status \"b5c85baeaf8e3bc4d53903705543dfe2296d252cd396456529c5676604641255\": rpc error: code = NotFound desc = could not find container \"b5c85baeaf8e3bc4d53903705543dfe2296d252cd396456529c5676604641255\": container with ID starting with b5c85baeaf8e3bc4d53903705543dfe2296d252cd396456529c5676604641255 not found: ID does not exist" Oct 31 13:31:47.206392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.206347 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" podStartSLOduration=2.206335605 podStartE2EDuration="2.206335605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:47.20492657 +0000 UTC m=+1675.350670417" watchObservedRunningTime="2026-10-31 13:31:47.206335605 +0000 UTC m=+1675.352079433" Oct 31 13:31:47.229889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.229866 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4"] Oct 31 13:31:47.239581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.239558 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c7b990ba-openshift-default-69444bf76f-m4gr4"] Oct 31 13:31:47.254305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.254274 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r"] Oct 31 13:31:47.259534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.259508 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-fc1bbe2b-gw-openshift-default-6fd96c6657-dqz8r"] Oct 31 13:31:47.368062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:47.367122 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h"] Oct 31 13:31:48.005215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:48.005182 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="204317d2-4400-4923-adac-e410dced28b3" path="/var/lib/kubelet/pods/204317d2-4400-4923-adac-e410dced28b3/volumes" Oct 31 13:31:48.005952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:48.005929 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" path="/var/lib/kubelet/pods/255e3805-e523-4620-b06a-f4d8fa91538f/volumes" Oct 31 13:31:50.038108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:50.038071 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj"] Oct 31 13:31:50.043103 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:50.043074 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5fb59233-xnzvj"] Oct 31 13:31:52.012349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:52.012312 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ea5bdf-96e9-4d3c-8d85-98494a327a40" path="/var/lib/kubelet/pods/56ea5bdf-96e9-4d3c-8d85-98494a327a40/volumes" Oct 31 13:31:52.368465 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:52.368389 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" podUID="0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c" containerName="payload-pre-processing" containerID="cri-o://3fb19e25cb94821b7924f2e39699157181ae274b833862de3bf4f3e8b63ed83a" gracePeriod=25 Oct 31 13:31:52.699723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:52.699633 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" Oct 31 13:31:52.788757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:52.788715 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-kube-api-access-brsfb\" (UniqueName: \"kubernetes.io/projected/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-kube-api-access-brsfb\") pod \"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c\" (UID: \"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c\") " Oct 31 13:31:52.789087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:52.788967 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-plugins-config-volume\") pod \"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c\" (UID: \"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c\") " Oct 31 13:31:52.790192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:52.790118 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-plugins-config-volume" pod "0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c" (UID: "0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:31:52.792666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:52.792638 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-kube-api-access-brsfb" pod "0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c" (UID: "0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c"). InnerVolumeSpecName "kube-api-access-brsfb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:31:52.891612 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:52.891562 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brsfb\" (UniqueName: \"kubernetes.io/projected/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-kube-api-access-brsfb\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:52.891612 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:52.891605 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:31:53.218795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:53.218763 5532 generic.go:356] "Generic (PLEG): container finished" podID="0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c" containerID="3fb19e25cb94821b7924f2e39699157181ae274b833862de3bf4f3e8b63ed83a" exitCode=0 Oct 31 13:31:53.219174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:53.218815 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" Oct 31 13:31:53.219174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:53.218849 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" event={"ID":"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c","Type":"ContainerDied","Data":"3fb19e25cb94821b7924f2e39699157181ae274b833862de3bf4f3e8b63ed83a"} Oct 31 13:31:53.219174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:53.218909 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h" event={"ID":"0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c","Type":"ContainerDied","Data":"e1be558e48db808bae37e4f146cda12bd30d4a057af84190942e85be77598c39"} Oct 31 13:31:53.219174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:53.218949 5532 scope.go:122] "RemoveContainer" containerID="3fb19e25cb94821b7924f2e39699157181ae274b833862de3bf4f3e8b63ed83a" Oct 31 13:31:53.230933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:53.230917 5532 scope.go:122] "RemoveContainer" containerID="3fb19e25cb94821b7924f2e39699157181ae274b833862de3bf4f3e8b63ed83a" Oct 31 13:31:53.231212 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:31:53.231194 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fb19e25cb94821b7924f2e39699157181ae274b833862de3bf4f3e8b63ed83a\": container with ID starting with 3fb19e25cb94821b7924f2e39699157181ae274b833862de3bf4f3e8b63ed83a not found: ID does not exist" containerID="3fb19e25cb94821b7924f2e39699157181ae274b833862de3bf4f3e8b63ed83a" Oct 31 13:31:53.231272 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:53.231222 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fb19e25cb94821b7924f2e39699157181ae274b833862de3bf4f3e8b63ed83a"} err="failed to get container status \"3fb19e25cb94821b7924f2e39699157181ae274b833862de3bf4f3e8b63ed83a\": rpc error: code = NotFound desc = could not find container \"3fb19e25cb94821b7924f2e39699157181ae274b833862de3bf4f3e8b63ed83a\": container with ID starting with 3fb19e25cb94821b7924f2e39699157181ae274b833862de3bf4f3e8b63ed83a not found: ID does not exist" Oct 31 13:31:53.253022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:53.252995 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h"] Oct 31 13:31:53.257848 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:53.257825 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a66ae960-b7f949887-nv98h"] Oct 31 13:31:54.004849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:54.004802 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c" path="/var/lib/kubelet/pods/0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c/volumes" Oct 31 13:31:54.077116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:54.077082 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6"] Oct 31 13:31:54.086126 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:54.086100 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97538cf4-g2wv6"] Oct 31 13:31:55.195833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.195798 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl"] Oct 31 13:31:55.215002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.214060 5532 cpu_manager.go:414] "containerMap: removing container" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" containerName="istio-proxy" Oct 31 13:31:55.215002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.214103 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" containerName="istio-proxy" Oct 31 13:31:55.215002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.214131 5532 cpu_manager.go:414] "containerMap: removing container" podUID="0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c" containerName="payload-pre-processing" Oct 31 13:31:55.215002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.214139 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c" containerName="payload-pre-processing" Oct 31 13:31:55.215002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.214161 5532 cpu_manager.go:414] "containerMap: removing container" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" Oct 31 13:31:55.215002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.214168 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" Oct 31 13:31:55.215002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.214685 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" containerName="istio-proxy" Oct 31 13:31:55.215002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.214717 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" containerName="istio-proxy" Oct 31 13:31:55.215002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.214734 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f1fc723-b9d7-4b78-a23e-e18df2aa2d5c" containerName="payload-pre-processing" Oct 31 13:31:55.215002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.214742 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" Oct 31 13:31:55.226987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.226952 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl"] Oct 31 13:31:55.227169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.227092 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.239232 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.239204 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-4f340c-serving-cert\"" Oct 31 13:31:55.333824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.333787 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.334111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.334089 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.334547 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.334519 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.334805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.334786 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7w7\" (UniqueName: \"kubernetes.io/projected/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-kube-api-access-gs7w7\") pod \"maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.437088 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.437049 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.437422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.437390 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7w7\" (UniqueName: \"kubernetes.io/projected/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-kube-api-access-gs7w7\") pod \"maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.437565 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.437511 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.437677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.437573 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.438523 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.438498 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.448124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.442156 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.448262 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.448134 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.459573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.459542 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7w7\" (UniqueName: \"kubernetes.io/projected/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-kube-api-access-gs7w7\") pod \"maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.557305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.557273 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:55.857570 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:31:55.857541 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed42dc6_80f5_4a58_9b3d_ffca597ddef5.slice/crio-3e60a61ce28b648f0ed26c35cd0a88742e30f22240e2def3cbae2f678692874d WatchSource:0}: Error finding container 3e60a61ce28b648f0ed26c35cd0a88742e30f22240e2def3cbae2f678692874d: Status 404 returned error can't find the container with id 3e60a61ce28b648f0ed26c35cd0a88742e30f22240e2def3cbae2f678692874d Oct 31 13:31:55.858945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.858917 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl"] Oct 31 13:31:55.859659 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:55.859638 5532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:31:56.031882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:56.031848 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a197f417-2bd8-4495-b4cb-07e4cfbd0eb0" path="/var/lib/kubelet/pods/a197f417-2bd8-4495-b4cb-07e4cfbd0eb0/volumes" Oct 31 13:31:56.249574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:56.249543 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" event={"ID":"aed42dc6-80f5-4a58-9b3d-ffca597ddef5","Type":"ContainerStarted","Data":"3e60a61ce28b648f0ed26c35cd0a88742e30f22240e2def3cbae2f678692874d"} Oct 31 13:31:56.487819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:56.487788 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:31:56.566263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:56.566173 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664"] Oct 31 13:31:56.633166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:56.626130 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b"] Oct 31 13:31:57.286653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:57.281404 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" event={"ID":"aed42dc6-80f5-4a58-9b3d-ffca597ddef5","Type":"ContainerStarted","Data":"e870d6c9245fb909fd6eac252a527ffe67173c30a94fcae75b22ce712becff5d"} Oct 31 13:31:57.286653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:57.283727 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:31:58.113776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:58.113711 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" podStartSLOduration=2.6676132580000003 podStartE2EDuration="3.113688304s" totalImagesPullingTime="446.075046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:31:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:31:57.323873237 +0000 UTC m=+1685.469617065" watchObservedRunningTime="2026-10-31 13:31:58.113688304 +0000 UTC m=+1686.259432133" Oct 31 13:31:58.114717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:58.114673 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z"] Oct 31 13:31:58.128428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:58.126160 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a66ae960-8jr5z"] Oct 31 13:31:58.174454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:58.174423 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" Oct 31 13:31:59.192230 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:31:59.190556 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" Oct 31 13:32:00.148620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:00.134737 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="685d6144-a686-4134-98a9-154bede4333b" path="/var/lib/kubelet/pods/685d6144-a686-4134-98a9-154bede4333b/volumes" Oct 31 13:32:01.049937 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:01.049876 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22"] Oct 31 13:32:01.116619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:01.109287 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf350ff9-2tv22"] Oct 31 13:32:01.576710 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:01.576664 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" podUID="f170a569-5b9f-4258-8456-6f3e195c4d46" containerName="payload-processing" containerID="cri-o://5f0f072eb3b53f65bb861e6c44957a02aa0fde8689fc08dd56301dc589e723cc" gracePeriod=25 Oct 31 13:32:01.627929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:01.627794 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" podUID="42886c31-8e0b-44e7-b1da-f3cddb479e92" containerName="payload-pre-processing" containerID="cri-o://38beb8b6dfff6f0b354ccbb00727a710d91e30e95b4f0cb192cc62a05944b1e9" gracePeriod=25 Oct 31 13:32:01.888579 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:32:01.888063 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:32:02.092882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.092843 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9792277-7cb7-47e7-9e0a-0f14d934058d" path="/var/lib/kubelet/pods/d9792277-7cb7-47e7-9e0a-0f14d934058d/volumes" Oct 31 13:32:02.112855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.107270 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj"] Oct 31 13:32:02.112855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.107319 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b86384-2p5bj"] Oct 31 13:32:02.248840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.248808 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" Oct 31 13:32:02.278118 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.278087 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" Oct 31 13:32:02.320994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.320935 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f170a569-5b9f-4258-8456-6f3e195c4d46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f170a569-5b9f-4258-8456-6f3e195c4d46-plugins-config-volume\") pod \"f170a569-5b9f-4258-8456-6f3e195c4d46\" (UID: \"f170a569-5b9f-4258-8456-6f3e195c4d46\") " Oct 31 13:32:02.321632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.321574 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f170a569-5b9f-4258-8456-6f3e195c4d46-plugins-config-volume" pod "f170a569-5b9f-4258-8456-6f3e195c4d46" (UID: "f170a569-5b9f-4258-8456-6f3e195c4d46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:02.422867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.422824 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f170a569-5b9f-4258-8456-6f3e195c4d46-kube-api-access-5rplq\" (UniqueName: \"kubernetes.io/projected/f170a569-5b9f-4258-8456-6f3e195c4d46-kube-api-access-5rplq\") pod \"f170a569-5b9f-4258-8456-6f3e195c4d46\" (UID: \"f170a569-5b9f-4258-8456-6f3e195c4d46\") " Oct 31 13:32:02.423052 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.423031 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42886c31-8e0b-44e7-b1da-f3cddb479e92-kube-api-access-qtrff\" (UniqueName: \"kubernetes.io/projected/42886c31-8e0b-44e7-b1da-f3cddb479e92-kube-api-access-qtrff\") pod \"42886c31-8e0b-44e7-b1da-f3cddb479e92\" (UID: \"42886c31-8e0b-44e7-b1da-f3cddb479e92\") " Oct 31 13:32:02.423179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.423159 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42886c31-8e0b-44e7-b1da-f3cddb479e92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42886c31-8e0b-44e7-b1da-f3cddb479e92-plugins-config-volume\") pod \"42886c31-8e0b-44e7-b1da-f3cddb479e92\" (UID: \"42886c31-8e0b-44e7-b1da-f3cddb479e92\") " Oct 31 13:32:02.423640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.423577 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42886c31-8e0b-44e7-b1da-f3cddb479e92-plugins-config-volume" pod "42886c31-8e0b-44e7-b1da-f3cddb479e92" (UID: "42886c31-8e0b-44e7-b1da-f3cddb479e92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:02.424133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.424107 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42886c31-8e0b-44e7-b1da-f3cddb479e92-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:02.424205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.424139 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f170a569-5b9f-4258-8456-6f3e195c4d46-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:02.425238 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.425212 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f170a569-5b9f-4258-8456-6f3e195c4d46-kube-api-access-5rplq" pod "f170a569-5b9f-4258-8456-6f3e195c4d46" (UID: "f170a569-5b9f-4258-8456-6f3e195c4d46"). InnerVolumeSpecName "kube-api-access-5rplq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:02.425476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.425448 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42886c31-8e0b-44e7-b1da-f3cddb479e92-kube-api-access-qtrff" pod "42886c31-8e0b-44e7-b1da-f3cddb479e92" (UID: "42886c31-8e0b-44e7-b1da-f3cddb479e92"). InnerVolumeSpecName "kube-api-access-qtrff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:02.445883 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.445804 5532 generic.go:356] "Generic (PLEG): container finished" podID="f170a569-5b9f-4258-8456-6f3e195c4d46" containerID="5f0f072eb3b53f65bb861e6c44957a02aa0fde8689fc08dd56301dc589e723cc" exitCode=0 Oct 31 13:32:02.446021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.445927 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" event={"ID":"f170a569-5b9f-4258-8456-6f3e195c4d46","Type":"ContainerDied","Data":"5f0f072eb3b53f65bb861e6c44957a02aa0fde8689fc08dd56301dc589e723cc"} Oct 31 13:32:02.446021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.445983 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" event={"ID":"f170a569-5b9f-4258-8456-6f3e195c4d46","Type":"ContainerDied","Data":"6f10ea934cc23bee6bf5a1c3b554d02d2e43d76e514b242ea8239f92b0b2f6d8"} Oct 31 13:32:02.446021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.446013 5532 scope.go:122] "RemoveContainer" containerID="5f0f072eb3b53f65bb861e6c44957a02aa0fde8689fc08dd56301dc589e723cc" Oct 31 13:32:02.446240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.446191 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664" Oct 31 13:32:02.448847 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.448814 5532 generic.go:356] "Generic (PLEG): container finished" podID="42886c31-8e0b-44e7-b1da-f3cddb479e92" containerID="38beb8b6dfff6f0b354ccbb00727a710d91e30e95b4f0cb192cc62a05944b1e9" exitCode=0 Oct 31 13:32:02.448957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.448938 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" event={"ID":"42886c31-8e0b-44e7-b1da-f3cddb479e92","Type":"ContainerDied","Data":"38beb8b6dfff6f0b354ccbb00727a710d91e30e95b4f0cb192cc62a05944b1e9"} Oct 31 13:32:02.449014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.448996 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" event={"ID":"42886c31-8e0b-44e7-b1da-f3cddb479e92","Type":"ContainerDied","Data":"84d2af85c37012b741e421c5e9837f4a25ee691712d952995fd53468aceb4c76"} Oct 31 13:32:02.449097 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.449072 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b" Oct 31 13:32:02.465075 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.464998 5532 scope.go:122] "RemoveContainer" containerID="5f0f072eb3b53f65bb861e6c44957a02aa0fde8689fc08dd56301dc589e723cc" Oct 31 13:32:02.465880 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:32:02.465476 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f0f072eb3b53f65bb861e6c44957a02aa0fde8689fc08dd56301dc589e723cc\": container with ID starting with 5f0f072eb3b53f65bb861e6c44957a02aa0fde8689fc08dd56301dc589e723cc not found: ID does not exist" containerID="5f0f072eb3b53f65bb861e6c44957a02aa0fde8689fc08dd56301dc589e723cc" Oct 31 13:32:02.465880 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.465517 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f0f072eb3b53f65bb861e6c44957a02aa0fde8689fc08dd56301dc589e723cc"} err="failed to get container status \"5f0f072eb3b53f65bb861e6c44957a02aa0fde8689fc08dd56301dc589e723cc\": rpc error: code = NotFound desc = could not find container \"5f0f072eb3b53f65bb861e6c44957a02aa0fde8689fc08dd56301dc589e723cc\": container with ID starting with 5f0f072eb3b53f65bb861e6c44957a02aa0fde8689fc08dd56301dc589e723cc not found: ID does not exist" Oct 31 13:32:02.465880 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.465542 5532 scope.go:122] "RemoveContainer" containerID="38beb8b6dfff6f0b354ccbb00727a710d91e30e95b4f0cb192cc62a05944b1e9" Oct 31 13:32:02.480540 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.480518 5532 scope.go:122] "RemoveContainer" containerID="38beb8b6dfff6f0b354ccbb00727a710d91e30e95b4f0cb192cc62a05944b1e9" Oct 31 13:32:02.480931 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:32:02.480899 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38beb8b6dfff6f0b354ccbb00727a710d91e30e95b4f0cb192cc62a05944b1e9\": container with ID starting with 38beb8b6dfff6f0b354ccbb00727a710d91e30e95b4f0cb192cc62a05944b1e9 not found: ID does not exist" containerID="38beb8b6dfff6f0b354ccbb00727a710d91e30e95b4f0cb192cc62a05944b1e9" Oct 31 13:32:02.481044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.480939 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38beb8b6dfff6f0b354ccbb00727a710d91e30e95b4f0cb192cc62a05944b1e9"} err="failed to get container status \"38beb8b6dfff6f0b354ccbb00727a710d91e30e95b4f0cb192cc62a05944b1e9\": rpc error: code = NotFound desc = could not find container \"38beb8b6dfff6f0b354ccbb00727a710d91e30e95b4f0cb192cc62a05944b1e9\": container with ID starting with 38beb8b6dfff6f0b354ccbb00727a710d91e30e95b4f0cb192cc62a05944b1e9 not found: ID does not exist" Oct 31 13:32:02.493738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.493687 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b"] Oct 31 13:32:02.501822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.501796 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-dff70365-ddf54c767-g6p6b"] Oct 31 13:32:02.521280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.521253 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664"] Oct 31 13:32:02.525238 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.525213 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtrff\" (UniqueName: \"kubernetes.io/projected/42886c31-8e0b-44e7-b1da-f3cddb479e92-kube-api-access-qtrff\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:02.525238 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.525239 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rplq\" (UniqueName: \"kubernetes.io/projected/f170a569-5b9f-4258-8456-6f3e195c4d46-kube-api-access-5rplq\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:02.529408 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:02.529379 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-dff70365-577bb65985-lv664"] Oct 31 13:32:04.006683 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:04.006645 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34b79093-ec53-40ab-a284-c7d05a4c61f0" path="/var/lib/kubelet/pods/34b79093-ec53-40ab-a284-c7d05a4c61f0/volumes" Oct 31 13:32:04.007313 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:04.007294 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42886c31-8e0b-44e7-b1da-f3cddb479e92" path="/var/lib/kubelet/pods/42886c31-8e0b-44e7-b1da-f3cddb479e92/volumes" Oct 31 13:32:04.007866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:04.007848 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f170a569-5b9f-4258-8456-6f3e195c4d46" path="/var/lib/kubelet/pods/f170a569-5b9f-4258-8456-6f3e195c4d46/volumes" Oct 31 13:32:04.037032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:04.037002 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4"] Oct 31 13:32:04.041658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:04.041636 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-589d748d-jplz4"] Oct 31 13:32:04.345349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:04.345263 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:32:06.005710 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:06.005679 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1cc949a-c1bb-41e7-8a85-465ff45fbfb2" path="/var/lib/kubelet/pods/b1cc949a-c1bb-41e7-8a85-465ff45fbfb2/volumes" Oct 31 13:32:06.824799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:06.824762 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb"] Oct 31 13:32:06.825006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:06.824985 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" podUID="5fe27f22-54e9-4332-aedc-3092b826b153" containerName="maas-api" containerID="cri-o://13d8b206b514dfec9705b9bd2b69fd80cbce25c3489bce7a79c954ac463d14e5" gracePeriod=30 Oct 31 13:32:07.313745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.313714 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:32:07.380205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.380169 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fe27f22-54e9-4332-aedc-3092b826b153-kube-api-access-2kjw7\" (UniqueName: \"kubernetes.io/projected/5fe27f22-54e9-4332-aedc-3092b826b153-kube-api-access-2kjw7\") pod \"5fe27f22-54e9-4332-aedc-3092b826b153\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " Oct 31 13:32:07.380403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.380242 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fe27f22-54e9-4332-aedc-3092b826b153-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe27f22-54e9-4332-aedc-3092b826b153-openshift-service-ca\") pod \"5fe27f22-54e9-4332-aedc-3092b826b153\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " Oct 31 13:32:07.380468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.380406 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-maas-api-tls\") pod \"5fe27f22-54e9-4332-aedc-3092b826b153\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " Oct 31 13:32:07.380572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.380555 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-metrics-certs\") pod \"5fe27f22-54e9-4332-aedc-3092b826b153\" (UID: \"5fe27f22-54e9-4332-aedc-3092b826b153\") " Oct 31 13:32:07.380712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.380682 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fe27f22-54e9-4332-aedc-3092b826b153-openshift-service-ca" pod "5fe27f22-54e9-4332-aedc-3092b826b153" (UID: "5fe27f22-54e9-4332-aedc-3092b826b153"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:32:07.381308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.381278 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5fe27f22-54e9-4332-aedc-3092b826b153-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:07.382871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.382846 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-maas-api-tls" pod "5fe27f22-54e9-4332-aedc-3092b826b153" (UID: "5fe27f22-54e9-4332-aedc-3092b826b153"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:32:07.383290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.383265 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fe27f22-54e9-4332-aedc-3092b826b153-kube-api-access-2kjw7" pod "5fe27f22-54e9-4332-aedc-3092b826b153" (UID: "5fe27f22-54e9-4332-aedc-3092b826b153"). InnerVolumeSpecName "kube-api-access-2kjw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:32:07.384298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.384272 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-metrics-certs" pod "5fe27f22-54e9-4332-aedc-3092b826b153" (UID: "5fe27f22-54e9-4332-aedc-3092b826b153"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:32:07.483074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.483039 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kjw7\" (UniqueName: \"kubernetes.io/projected/5fe27f22-54e9-4332-aedc-3092b826b153-kube-api-access-2kjw7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:07.483074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.483071 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:07.483074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.483081 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5fe27f22-54e9-4332-aedc-3092b826b153-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:32:07.543908 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.543798 5532 generic.go:356] "Generic (PLEG): container finished" podID="5fe27f22-54e9-4332-aedc-3092b826b153" containerID="13d8b206b514dfec9705b9bd2b69fd80cbce25c3489bce7a79c954ac463d14e5" exitCode=0 Oct 31 13:32:07.543908 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.543857 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" Oct 31 13:32:07.543908 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.543881 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" event={"ID":"5fe27f22-54e9-4332-aedc-3092b826b153","Type":"ContainerDied","Data":"13d8b206b514dfec9705b9bd2b69fd80cbce25c3489bce7a79c954ac463d14e5"} Oct 31 13:32:07.544175 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.543951 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb" event={"ID":"5fe27f22-54e9-4332-aedc-3092b826b153","Type":"ContainerDied","Data":"d4cab0507f8b06ec8dea1047fc846422c1ee80482aa02baa7b801a71f6cf8599"} Oct 31 13:32:07.544175 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.543970 5532 scope.go:122] "RemoveContainer" containerID="13d8b206b514dfec9705b9bd2b69fd80cbce25c3489bce7a79c954ac463d14e5" Oct 31 13:32:07.557726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.557684 5532 scope.go:122] "RemoveContainer" containerID="13d8b206b514dfec9705b9bd2b69fd80cbce25c3489bce7a79c954ac463d14e5" Oct 31 13:32:07.558005 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:32:07.557986 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d8b206b514dfec9705b9bd2b69fd80cbce25c3489bce7a79c954ac463d14e5\": container with ID starting with 13d8b206b514dfec9705b9bd2b69fd80cbce25c3489bce7a79c954ac463d14e5 not found: ID does not exist" containerID="13d8b206b514dfec9705b9bd2b69fd80cbce25c3489bce7a79c954ac463d14e5" Oct 31 13:32:07.558067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.558015 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d8b206b514dfec9705b9bd2b69fd80cbce25c3489bce7a79c954ac463d14e5"} err="failed to get container status \"13d8b206b514dfec9705b9bd2b69fd80cbce25c3489bce7a79c954ac463d14e5\": rpc error: code = NotFound desc = could not find container \"13d8b206b514dfec9705b9bd2b69fd80cbce25c3489bce7a79c954ac463d14e5\": container with ID starting with 13d8b206b514dfec9705b9bd2b69fd80cbce25c3489bce7a79c954ac463d14e5 not found: ID does not exist" Oct 31 13:32:07.579263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.579230 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb"] Oct 31 13:32:07.584802 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:07.584771 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-dff70365-5745fb6488-8pmzb"] Oct 31 13:32:08.050626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:08.047485 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fe27f22-54e9-4332-aedc-3092b826b153" path="/var/lib/kubelet/pods/5fe27f22-54e9-4332-aedc-3092b826b153/volumes" Oct 31 13:32:09.057286 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:09.057246 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x"] Oct 31 13:32:09.061964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:09.061936 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e7a530-6vp6x"] Oct 31 13:32:10.006298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:10.006265 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c83ddeaf-165a-4f00-9a9a-2ac8bce136ca" path="/var/lib/kubelet/pods/c83ddeaf-165a-4f00-9a9a-2ac8bce136ca/volumes" Oct 31 13:32:15.030322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.030290 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr"] Oct 31 13:32:15.034742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.034716 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b4fb8bcb-r8kdr"] Oct 31 13:32:15.819258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.819225 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw"] Oct 31 13:32:15.819977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.819962 5532 cpu_manager.go:414] "containerMap: removing container" podUID="5fe27f22-54e9-4332-aedc-3092b826b153" containerName="maas-api" Oct 31 13:32:15.820033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.819979 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fe27f22-54e9-4332-aedc-3092b826b153" containerName="maas-api" Oct 31 13:32:15.820033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.819997 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f170a569-5b9f-4258-8456-6f3e195c4d46" containerName="payload-processing" Oct 31 13:32:15.820033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.820002 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f170a569-5b9f-4258-8456-6f3e195c4d46" containerName="payload-processing" Oct 31 13:32:15.820033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.820015 5532 cpu_manager.go:414] "containerMap: removing container" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" containerName="istio-proxy" Oct 31 13:32:15.820033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.820021 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="255e3805-e523-4620-b06a-f4d8fa91538f" containerName="istio-proxy" Oct 31 13:32:15.820033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.820029 5532 cpu_manager.go:414] "containerMap: removing container" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" Oct 31 13:32:15.820033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.820034 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" Oct 31 13:32:15.820240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.820075 5532 cpu_manager.go:414] "containerMap: removing container" podUID="42886c31-8e0b-44e7-b1da-f3cddb479e92" containerName="payload-pre-processing" Oct 31 13:32:15.820240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.820080 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42886c31-8e0b-44e7-b1da-f3cddb479e92" containerName="payload-pre-processing" Oct 31 13:32:15.820240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.820177 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="204317d2-4400-4923-adac-e410dced28b3" containerName="istio-proxy" Oct 31 13:32:15.820240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.820193 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="42886c31-8e0b-44e7-b1da-f3cddb479e92" containerName="payload-pre-processing" Oct 31 13:32:15.820240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.820203 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fe27f22-54e9-4332-aedc-3092b826b153" containerName="maas-api" Oct 31 13:32:15.820240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.820213 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f170a569-5b9f-4258-8456-6f3e195c4d46" containerName="payload-processing" Oct 31 13:32:15.824146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.824123 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.830034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.830011 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-b06b08-openshift-default-dockercfg-qvpwj\"" Oct 31 13:32:15.852533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.852504 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw"] Oct 31 13:32:15.866116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.866056 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-socket\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.866279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.866182 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-envoy\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.866279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.866262 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-credential-socket\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.866416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.866305 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-data\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.866416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.866358 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-certs\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.866532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.866501 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n8f8\" (UniqueName: \"kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-kube-api-access-4n8f8\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.866672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.866648 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-podinfo\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.866842 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.866818 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istiod-ca-cert\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.866936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.866918 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-token\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.973782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.973705 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istiod-ca-cert\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.974131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.973844 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-token\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.974131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.973924 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-socket\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.974131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.974006 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-envoy\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.974131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.974069 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-credential-socket\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.974131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.974130 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-data\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.974717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.974207 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-certs\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.974717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.974321 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n8f8\" (UniqueName: \"kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-kube-api-access-4n8f8\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.974717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.974423 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-podinfo\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.977916 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.977881 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-envoy\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.979468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.978555 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istiod-ca-cert\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.979468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.979133 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-socket\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.979468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.979318 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-podinfo\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.979468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.979369 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-data\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.980286 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.980135 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-credential-socket\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:15.984248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:15.984189 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-certs\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:16.009631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:16.000235 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-token\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:16.043638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:16.041342 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n8f8\" (UniqueName: \"kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-kube-api-access-4n8f8\") pod \"e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:16.060249 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:16.060219 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d9a5c2f-4082-421d-99b2-3169fd3b4c07" path="/var/lib/kubelet/pods/6d9a5c2f-4082-421d-99b2-3169fd3b4c07/volumes" Oct 31 13:32:16.150170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:16.150131 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:16.535874 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:32:16.535843 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b8b3d6_33ce_4ddf_a4ae_64a1d573c49d.slice/crio-fa8dfd059f174d7d4f32541d99c923de70c12fb9f8f0ec3c25187f69e059ae6c WatchSource:0}: Error finding container fa8dfd059f174d7d4f32541d99c923de70c12fb9f8f0ec3c25187f69e059ae6c: Status 404 returned error can't find the container with id fa8dfd059f174d7d4f32541d99c923de70c12fb9f8f0ec3c25187f69e059ae6c Oct 31 13:32:16.536181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:16.536156 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw"] Oct 31 13:32:16.538439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:16.538407 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:32:16.538525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:16.538468 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:32:16.538525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:16.538496 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:32:16.630494 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:16.630455 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" event={"ID":"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d","Type":"ContainerStarted","Data":"965f8963eb8cb9c73c2014686e459ee16690946c6eed7444a93069b411636bdb"} Oct 31 13:32:16.630658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:16.630640 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" event={"ID":"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d","Type":"ContainerStarted","Data":"fa8dfd059f174d7d4f32541d99c923de70c12fb9f8f0ec3c25187f69e059ae6c"} Oct 31 13:32:16.665169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:16.665080 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" podStartSLOduration=1.665058827 podStartE2EDuration="1.665058827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:16.656928521 +0000 UTC m=+1704.802672351" watchObservedRunningTime="2026-10-31 13:32:16.665058827 +0000 UTC m=+1704.810802660" Oct 31 13:32:17.150920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:17.150885 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:17.156603 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:17.156574 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:17.640492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:17.640405 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:17.641508 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:17.641489 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:32:19.072617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:19.072208 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6"] Oct 31 13:32:19.097616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:19.097518 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22dbacb2-h68k6"] Oct 31 13:32:20.005110 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.005078 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c258c3-2446-4fab-aa37-50b687f9651b" path="/var/lib/kubelet/pods/d3c258c3-2446-4fab-aa37-50b687f9651b/volumes" Oct 31 13:32:20.636362 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.636321 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6"] Oct 31 13:32:20.653234 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.653192 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4"] Oct 31 13:32:20.653559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.653536 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" Oct 31 13:32:20.658451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.658407 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" Oct 31 13:32:20.667844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.667634 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-b06b08\"" Oct 31 13:32:20.674717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.674690 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4"] Oct 31 13:32:20.683940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.683911 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-b06b08-dockercfg-k7mdj\"" Oct 31 13:32:20.692515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.692482 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6"] Oct 31 13:32:20.829617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.813836 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvzlb\" (UniqueName: \"kubernetes.io/projected/614f22b9-7512-481c-b8e3-f4757bca9265-kube-api-access-vvzlb\") pod \"payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4\" (UID: \"614f22b9-7512-481c-b8e3-f4757bca9265\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" Oct 31 13:32:20.829617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.814029 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrtfv\" (UniqueName: \"kubernetes.io/projected/6cc022d9-8637-426b-b8fe-6eeafb3bf823-kube-api-access-mrtfv\") pod \"payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6\" (UID: \"6cc022d9-8637-426b-b8fe-6eeafb3bf823\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" Oct 31 13:32:20.829617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.814222 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614f22b9-7512-481c-b8e3-f4757bca9265-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4\" (UID: \"614f22b9-7512-481c-b8e3-f4757bca9265\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" Oct 31 13:32:20.829617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.814330 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc022d9-8637-426b-b8fe-6eeafb3bf823-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6\" (UID: \"6cc022d9-8637-426b-b8fe-6eeafb3bf823\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" Oct 31 13:32:20.919618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.918733 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614f22b9-7512-481c-b8e3-f4757bca9265-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4\" (UID: \"614f22b9-7512-481c-b8e3-f4757bca9265\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" Oct 31 13:32:20.919618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.918832 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc022d9-8637-426b-b8fe-6eeafb3bf823-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6\" (UID: \"6cc022d9-8637-426b-b8fe-6eeafb3bf823\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" Oct 31 13:32:20.919618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.919011 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvzlb\" (UniqueName: \"kubernetes.io/projected/614f22b9-7512-481c-b8e3-f4757bca9265-kube-api-access-vvzlb\") pod \"payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4\" (UID: \"614f22b9-7512-481c-b8e3-f4757bca9265\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" Oct 31 13:32:20.919618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.919137 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrtfv\" (UniqueName: \"kubernetes.io/projected/6cc022d9-8637-426b-b8fe-6eeafb3bf823-kube-api-access-mrtfv\") pod \"payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6\" (UID: \"6cc022d9-8637-426b-b8fe-6eeafb3bf823\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" Oct 31 13:32:20.921929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.920559 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614f22b9-7512-481c-b8e3-f4757bca9265-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4\" (UID: \"614f22b9-7512-481c-b8e3-f4757bca9265\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" Oct 31 13:32:20.921929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.921325 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc022d9-8637-426b-b8fe-6eeafb3bf823-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6\" (UID: \"6cc022d9-8637-426b-b8fe-6eeafb3bf823\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" Oct 31 13:32:20.952708 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.952669 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvzlb\" (UniqueName: \"kubernetes.io/projected/614f22b9-7512-481c-b8e3-f4757bca9265-kube-api-access-vvzlb\") pod \"payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4\" (UID: \"614f22b9-7512-481c-b8e3-f4757bca9265\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" Oct 31 13:32:20.959451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.959409 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrtfv\" (UniqueName: \"kubernetes.io/projected/6cc022d9-8637-426b-b8fe-6eeafb3bf823-kube-api-access-mrtfv\") pod \"payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6\" (UID: \"6cc022d9-8637-426b-b8fe-6eeafb3bf823\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" Oct 31 13:32:20.974877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.974841 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" Oct 31 13:32:20.993261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:20.991213 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" Oct 31 13:32:21.281004 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:21.280974 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6"] Oct 31 13:32:21.308172 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:32:21.308144 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614f22b9_7512_481c_b8e3_f4757bca9265.slice/crio-54d2be61934a0d85e371ac3fdf35cad57595661bb1f15f1e287c6eb49a84016e WatchSource:0}: Error finding container 54d2be61934a0d85e371ac3fdf35cad57595661bb1f15f1e287c6eb49a84016e: Status 404 returned error can't find the container with id 54d2be61934a0d85e371ac3fdf35cad57595661bb1f15f1e287c6eb49a84016e Oct 31 13:32:21.310701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:21.310673 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4"] Oct 31 13:32:21.683379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:21.683335 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" event={"ID":"614f22b9-7512-481c-b8e3-f4757bca9265","Type":"ContainerStarted","Data":"2ac91c2461f838df11f47a1041d4ef4e44e25769e87f07bbc0aca267546c4f85"} Oct 31 13:32:21.683879 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:21.683408 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" event={"ID":"614f22b9-7512-481c-b8e3-f4757bca9265","Type":"ContainerStarted","Data":"54d2be61934a0d85e371ac3fdf35cad57595661bb1f15f1e287c6eb49a84016e"} Oct 31 13:32:21.683879 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:21.683436 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" Oct 31 13:32:21.685555 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:21.685522 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" event={"ID":"6cc022d9-8637-426b-b8fe-6eeafb3bf823","Type":"ContainerStarted","Data":"b1df64ab6cc1fa008391dda524cad636852b4e3facca3f42d712f2f67a3cfb51"} Oct 31 13:32:21.685759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:21.685580 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" Oct 31 13:32:21.685759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:21.685643 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" event={"ID":"6cc022d9-8637-426b-b8fe-6eeafb3bf823","Type":"ContainerStarted","Data":"aeda4143e7502541419b7fd0c069c71b35bf7e1c55ff3401c567677f9c08b86f"} Oct 31 13:32:21.724450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:21.724385 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" podStartSLOduration=1.724366293 podStartE2EDuration="1.724366293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:21.722178953 +0000 UTC m=+1709.867922782" watchObservedRunningTime="2026-10-31 13:32:21.724366293 +0000 UTC m=+1709.870110122" Oct 31 13:32:21.759928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:21.759862 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" podStartSLOduration=1.759839279 podStartE2EDuration="1.759839279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:21.748428544 +0000 UTC m=+1709.894172375" watchObservedRunningTime="2026-10-31 13:32:21.759839279 +0000 UTC m=+1709.905583107" Oct 31 13:32:24.037111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:24.037079 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8"] Oct 31 13:32:24.046640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:24.046578 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f107cfb-zbrz8"] Oct 31 13:32:26.018660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.018623 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41cb130e-a0d3-44dc-9e50-bc571de0c55f" path="/var/lib/kubelet/pods/41cb130e-a0d3-44dc-9e50-bc571de0c55f/volumes" Oct 31 13:32:26.692361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.692319 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4"] Oct 31 13:32:26.698559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.698535 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" Oct 31 13:32:26.707352 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.706111 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-79f4fb0f\"" Oct 31 13:32:26.710371 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.710343 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-79f4fb0f-dockercfg-4ks8x\"" Oct 31 13:32:26.713641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.712657 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4"] Oct 31 13:32:26.738686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.738653 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh84x\" (UniqueName: \"kubernetes.io/projected/ededb5f2-22bb-4d1f-896a-63c8cc779422-kube-api-access-hh84x\") pod \"payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4\" (UID: \"ededb5f2-22bb-4d1f-896a-63c8cc779422\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" Oct 31 13:32:26.738883 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.738826 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ededb5f2-22bb-4d1f-896a-63c8cc779422-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4\" (UID: \"ededb5f2-22bb-4d1f-896a-63c8cc779422\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" Oct 31 13:32:26.758612 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.758571 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt"] Oct 31 13:32:26.763621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.763579 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" Oct 31 13:32:26.784279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.784248 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt"] Oct 31 13:32:26.886665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.846699 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh84x\" (UniqueName: \"kubernetes.io/projected/ededb5f2-22bb-4d1f-896a-63c8cc779422-kube-api-access-hh84x\") pod \"payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4\" (UID: \"ededb5f2-22bb-4d1f-896a-63c8cc779422\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" Oct 31 13:32:26.887218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.887178 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ededb5f2-22bb-4d1f-896a-63c8cc779422-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4\" (UID: \"ededb5f2-22bb-4d1f-896a-63c8cc779422\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" Oct 31 13:32:26.908618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.888454 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ededb5f2-22bb-4d1f-896a-63c8cc779422-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4\" (UID: \"ededb5f2-22bb-4d1f-896a-63c8cc779422\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" Oct 31 13:32:27.004714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.997248 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh84x\" (UniqueName: \"kubernetes.io/projected/ededb5f2-22bb-4d1f-896a-63c8cc779422-kube-api-access-hh84x\") pod \"payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4\" (UID: \"ededb5f2-22bb-4d1f-896a-63c8cc779422\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" Oct 31 13:32:27.004714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.998472 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt\" (UID: \"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2\") " pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" Oct 31 13:32:27.004714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:26.999532 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrrvk\" (UniqueName: \"kubernetes.io/projected/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-kube-api-access-hrrvk\") pod \"payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt\" (UID: \"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2\") " pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" Oct 31 13:32:27.016570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.016543 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" Oct 31 13:32:27.101817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.101786 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrrvk\" (UniqueName: \"kubernetes.io/projected/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-kube-api-access-hrrvk\") pod \"payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt\" (UID: \"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2\") " pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" Oct 31 13:32:27.102249 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.101913 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt\" (UID: \"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2\") " pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" Oct 31 13:32:27.104819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.102675 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-plugins-config-volume\") pod \"payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt\" (UID: \"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2\") " pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" Oct 31 13:32:27.120392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.117975 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrrvk\" (UniqueName: \"kubernetes.io/projected/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-kube-api-access-hrrvk\") pod \"payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt\" (UID: \"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2\") " pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" Oct 31 13:32:27.129229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.128938 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" Oct 31 13:32:27.353703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.352851 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4"] Oct 31 13:32:27.363089 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:32:27.362983 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podededb5f2_22bb_4d1f_896a_63c8cc779422.slice/crio-5ad20d4548f6f3be7bc40ed24d89b74a46debaf887293c5f1d8c8853733c2d90 WatchSource:0}: Error finding container 5ad20d4548f6f3be7bc40ed24d89b74a46debaf887293c5f1d8c8853733c2d90: Status 404 returned error can't find the container with id 5ad20d4548f6f3be7bc40ed24d89b74a46debaf887293c5f1d8c8853733c2d90 Oct 31 13:32:27.446509 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:32:27.446478 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f5d90d8_494d_4be6_81a8_d8fc27e5f8d2.slice/crio-803b288e8cafd9f4f5f55c2c41e48550d04450e10699c5038c5b262f40782a0c WatchSource:0}: Error finding container 803b288e8cafd9f4f5f55c2c41e48550d04450e10699c5038c5b262f40782a0c: Status 404 returned error can't find the container with id 803b288e8cafd9f4f5f55c2c41e48550d04450e10699c5038c5b262f40782a0c Oct 31 13:32:27.447126 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.447101 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt"] Oct 31 13:32:27.750457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.750422 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" event={"ID":"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2","Type":"ContainerStarted","Data":"b32b3ab39d686004880790adf10c37dda14d770f2dd40e659e970b36d566ae50"} Oct 31 13:32:27.750689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.750489 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" Oct 31 13:32:27.750689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.750517 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" event={"ID":"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2","Type":"ContainerStarted","Data":"803b288e8cafd9f4f5f55c2c41e48550d04450e10699c5038c5b262f40782a0c"} Oct 31 13:32:27.752252 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.752223 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" event={"ID":"ededb5f2-22bb-4d1f-896a-63c8cc779422","Type":"ContainerStarted","Data":"d2b7d41ae02f7da641319599873650ee5a64ea18a9531ce5b3b6747c305b9405"} Oct 31 13:32:27.752353 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.752268 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" event={"ID":"ededb5f2-22bb-4d1f-896a-63c8cc779422","Type":"ContainerStarted","Data":"5ad20d4548f6f3be7bc40ed24d89b74a46debaf887293c5f1d8c8853733c2d90"} Oct 31 13:32:27.752353 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.752300 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" Oct 31 13:32:27.790889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.790838 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" podStartSLOduration=1.790824362 podStartE2EDuration="1.790824362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:27.788323927 +0000 UTC m=+1715.934067755" watchObservedRunningTime="2026-10-31 13:32:27.790824362 +0000 UTC m=+1715.936568208" Oct 31 13:32:27.853343 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:27.853282 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" podStartSLOduration=1.853265411 podStartE2EDuration="1.853265411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:27.852875989 +0000 UTC m=+1715.998619821" watchObservedRunningTime="2026-10-31 13:32:27.853265411 +0000 UTC m=+1715.999009241" Oct 31 13:32:28.181033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.181000 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99"] Oct 31 13:32:28.197675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.197639 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:28.214093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.214050 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-b06b08-serving-cert\"" Oct 31 13:32:28.222701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.222671 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4caf91-ff54-45c0-98a1-7bbb51abb628-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:28.222863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.222817 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:28.224541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.224434 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:28.224857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.224829 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z85wm\" (UniqueName: \"kubernetes.io/projected/da4caf91-ff54-45c0-98a1-7bbb51abb628-kube-api-access-z85wm\") pod \"maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:28.226584 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.226369 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99"] Oct 31 13:32:28.326235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.326157 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:28.327299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.327243 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:28.327716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.327694 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z85wm\" (UniqueName: \"kubernetes.io/projected/da4caf91-ff54-45c0-98a1-7bbb51abb628-kube-api-access-z85wm\") pod \"maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:28.328504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.328474 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4caf91-ff54-45c0-98a1-7bbb51abb628-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:28.329504 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:32:28.327561 5532 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-serving-cert: secret "maas-api-e2e-shared-b-w2-b06b08-serving-cert" not found Oct 31 13:32:28.329629 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:32:28.329563 5532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-maas-api-tls podName:da4caf91-ff54-45c0-98a1-7bbb51abb628 nodeName:}" failed. No retries permitted until 2026-10-31 13:32:28.82953898 +0000 UTC m=+1716.975282798 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-maas-api-tls") pod "maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" (UID: "da4caf91-ff54-45c0-98a1-7bbb51abb628") : secret "maas-api-e2e-shared-b-w2-b06b08-serving-cert" not found Oct 31 13:32:28.329759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.329355 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4caf91-ff54-45c0-98a1-7bbb51abb628-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:28.333180 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.333036 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:28.349714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.349681 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z85wm\" (UniqueName: \"kubernetes.io/projected/da4caf91-ff54-45c0-98a1-7bbb51abb628-kube-api-access-z85wm\") pod \"maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:28.839470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.839437 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:28.842050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:28.842020 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:29.137196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:29.137113 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:29.522026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:29.521989 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99"] Oct 31 13:32:29.779137 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:29.779018 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" event={"ID":"da4caf91-ff54-45c0-98a1-7bbb51abb628","Type":"ContainerStarted","Data":"8a8273d74f4fbc1afb78bd3b17a3442880086f06793d36be95f4036b0904708d"} Oct 31 13:32:30.789795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:30.789745 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" event={"ID":"da4caf91-ff54-45c0-98a1-7bbb51abb628","Type":"ContainerStarted","Data":"a198217546876a0e2160fde142b252e7f1c25cedb6256bb2dec705095c7bcd69"} Oct 31 13:32:30.790191 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:30.789831 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:30.836108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:30.836045 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" podStartSLOduration=2.453705893 podStartE2EDuration="2.836027153s" totalImagesPullingTime="382.32126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:30.830126482 +0000 UTC m=+1718.975870309" watchObservedRunningTime="2026-10-31 13:32:30.836027153 +0000 UTC m=+1718.981770984" Oct 31 13:32:31.769510 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:32:31.769466 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:32:32.042908 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:32.042824 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272"] Oct 31 13:32:32.047643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:32.047614 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9778441d-bq272"] Oct 31 13:32:32.698164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:32.698133 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" Oct 31 13:32:32.698368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:32.698230 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" Oct 31 13:32:34.034667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:34.022321 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4121a829-90af-4108-ac5c-d2aef1e797c8" path="/var/lib/kubelet/pods/4121a829-90af-4108-ac5c-d2aef1e797c8/volumes" Oct 31 13:32:36.661779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.655728 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z"] Oct 31 13:32:36.730758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.730699 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:32:36.738349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.738315 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z"] Oct 31 13:32:36.738731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.738709 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:36.757179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.757140 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-metrics-certs\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:36.757500 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.757476 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-openshift-service-ca\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:36.757831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.757809 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-maas-api-tls\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:36.758041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.758023 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnqm7\" (UniqueName: \"kubernetes.io/projected/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-kube-api-access-dnqm7\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:36.779964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.779042 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-79f4fb0f-serving-cert\"" Oct 31 13:32:36.841957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.839893 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:36.844118 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.843796 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:32:36.877895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.864897 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnqm7\" (UniqueName: \"kubernetes.io/projected/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-kube-api-access-dnqm7\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:36.877895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.865082 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-metrics-certs\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:36.877895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.865210 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-openshift-service-ca\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:36.877895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.865354 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-maas-api-tls\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:36.877895 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:32:36.865570 5532 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-serving-cert: secret "maas-api-e2e-mt-79f4fb0f-serving-cert" not found Oct 31 13:32:36.877895 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:32:36.865656 5532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-maas-api-tls podName:c4ebfc88-1f62-43e9-bbf9-96216efffbe7 nodeName:}" failed. No retries permitted until 2026-10-31 13:32:37.365633812 +0000 UTC m=+1725.511377621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-maas-api-tls") pod "maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" (UID: "c4ebfc88-1f62-43e9-bbf9-96216efffbe7") : secret "maas-api-e2e-mt-79f4fb0f-serving-cert" not found Oct 31 13:32:36.877895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.876940 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-openshift-service-ca\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:36.931678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.910707 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:32:36.931678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.923985 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-metrics-certs\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:36.931678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:36.924838 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:37.057625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.035969 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:37.087049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.063846 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnqm7\" (UniqueName: \"kubernetes.io/projected/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-kube-api-access-dnqm7\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:37.087049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.070397 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:37.204181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.190720 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:37.204181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.191795 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:37.369418 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.369171 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-maas-api-tls\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:37.369644 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:32:37.369526 5532 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-serving-cert: secret "maas-api-e2e-mt-79f4fb0f-serving-cert" not found Oct 31 13:32:37.369644 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:32:37.369643 5532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-maas-api-tls podName:c4ebfc88-1f62-43e9-bbf9-96216efffbe7 nodeName:}" failed. No retries permitted until 2026-10-31 13:32:38.369571828 +0000 UTC m=+1726.515315648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-maas-api-tls") pod "maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" (UID: "c4ebfc88-1f62-43e9-bbf9-96216efffbe7") : secret "maas-api-e2e-mt-79f4fb0f-serving-cert" not found Oct 31 13:32:37.451658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.447102 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:37.459882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.459778 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:37.463618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.460416 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:32:37.527619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.523927 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 31 13:32:37.735007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.734893 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:32:37.748619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.736053 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:37.748619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.746433 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:32:37.917559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.917509 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:37.918301 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:37.918271 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:32:38.077183 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.071019 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:38.134075 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.134031 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:38.199780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.176209 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:32:38.217823 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.204133 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:32:38.217823 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.204463 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:32:38.473581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.473375 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:38.481439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.481406 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:32:38.503739 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.503709 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:32:38.504013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.503996 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:32:38.504150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.504138 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:32:38.504283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.504269 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:32:38.504423 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.504409 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:32:38.504551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.504539 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:32:38.504695 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.504682 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:32:38.504830 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.504817 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:32:38.524150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.494830 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-maas-api-tls\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:38.525876 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.525851 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:32:38.526102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.526089 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:32:38.533318 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.533294 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:32:38.615221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.615168 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-maas-api-tls\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:38.691021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.690935 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:38.780331 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.779269 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" Oct 31 13:32:38.781905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:38.781873 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" Oct 31 13:32:39.074333 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:32:39.074284 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ebfc88_1f62_43e9_bbf9_96216efffbe7.slice/crio-ae259a13216c97927d2caf7cd1ab37b6701e843a098b44d42069534236cae187 WatchSource:0}: Error finding container ae259a13216c97927d2caf7cd1ab37b6701e843a098b44d42069534236cae187: Status 404 returned error can't find the container with id ae259a13216c97927d2caf7cd1ab37b6701e843a098b44d42069534236cae187 Oct 31 13:32:39.076101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:39.076071 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z"] Oct 31 13:32:39.575187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:39.575154 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" event={"ID":"c4ebfc88-1f62-43e9-bbf9-96216efffbe7","Type":"ContainerStarted","Data":"ae259a13216c97927d2caf7cd1ab37b6701e843a098b44d42069534236cae187"} Oct 31 13:32:40.585960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:40.585921 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" event={"ID":"c4ebfc88-1f62-43e9-bbf9-96216efffbe7","Type":"ContainerStarted","Data":"bb9502d420adf0cb85a93db5b3f9f683ef4a18548d940fbc5c0b436f020540fd"} Oct 31 13:32:40.586477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:40.586007 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:40.617691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:40.617641 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" podStartSLOduration=5.184085515 podStartE2EDuration="5.617627923s" totalImagesPullingTime="433.542408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:32:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:32:40.611324703 +0000 UTC m=+1728.757068531" watchObservedRunningTime="2026-10-31 13:32:40.617627923 +0000 UTC m=+1728.763371747" Oct 31 13:32:42.065604 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:42.065557 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k"] Oct 31 13:32:42.066007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:42.065618 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f9812a60-xp55k"] Oct 31 13:32:44.006221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:44.006169 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f16a2544-082d-465a-86cc-709a38c3adc1" path="/var/lib/kubelet/pods/f16a2544-082d-465a-86cc-709a38c3adc1/volumes" Oct 31 13:32:44.093759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:44.093711 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:32:44.093910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:44.093777 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:32:45.701253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:45.701221 5532 generic.go:356] "Generic (PLEG): container finished" podID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerID="aac1668030455d43982c62c2232c93ad7105a4b718e8b8430cd35d2636767b9b" exitCode=0 Oct 31 13:32:45.701713 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:45.701304 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" event={"ID":"a3dd044e-3702-46d9-91db-2d456c9dd345","Type":"ContainerDied","Data":"aac1668030455d43982c62c2232c93ad7105a4b718e8b8430cd35d2636767b9b"} Oct 31 13:32:45.701824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:45.701807 5532 scope.go:122] "RemoveContainer" containerID="aac1668030455d43982c62c2232c93ad7105a4b718e8b8430cd35d2636767b9b" Oct 31 13:32:45.703502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:45.703476 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:32:45.703574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:45.703547 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:32:45.703646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:45.703577 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:32:46.664199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:46.664154 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:32:46.713938 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:46.713359 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" event={"ID":"a3dd044e-3702-46d9-91db-2d456c9dd345","Type":"ContainerStarted","Data":"31773840c162b2e52d89cfea61fdfb82a72d67685ef86fc52f06ef4b91311f10"} Oct 31 13:32:47.701995 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:47.701963 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:32:47.702170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:47.702127 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 13:32:47.702228 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:47.702171 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 13:32:48.701473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:48.701435 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 13:32:48.701904 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:48.701516 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 13:32:49.705057 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:49.705016 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 13:32:49.705680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:49.705645 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 13:32:50.701479 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:50.701442 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 13:32:50.701693 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:50.701503 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 13:32:51.702095 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:51.702061 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Oct 31 13:32:51.702511 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:51.702111 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Oct 31 13:32:52.321655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:52.321624 5532 scope.go:122] "RemoveContainer" containerID="16b15ca4d55c41127e10c994ae81f968bd9559f6da1ac74d84cb9cc05b32170d" Oct 31 13:32:52.333029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:52.333007 5532 scope.go:122] "RemoveContainer" containerID="e126f3d454a5890579fe0e6d98686c09f026b207260da7a06c5aad2c04b2ba50" Oct 31 13:32:52.342748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:52.342732 5532 scope.go:122] "RemoveContainer" containerID="dd0d2558d2a22bdcaea06f01d3a911c5c7e9c8aae340242dc56590906da805d0" Oct 31 13:32:52.705963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:52.705935 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:32:52.706469 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:52.706446 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:32:52.707224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:52.707204 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:32:55.032381 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:55.032348 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s"] Oct 31 13:32:55.037372 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:55.037345 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-737725b5-ph78s"] Oct 31 13:32:56.005750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:32:56.005712 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ea0fbcc-2d0c-415a-a5cf-e3f8075737de" path="/var/lib/kubelet/pods/3ea0fbcc-2d0c-415a-a5cf-e3f8075737de/volumes" Oct 31 13:33:01.763786 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:33:01.763749 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:33:02.712949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:02.712918 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:33:15.075683 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:15.075574 5532 generic.go:356] "Generic (PLEG): container finished" podID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerID="965f8963eb8cb9c73c2014686e459ee16690946c6eed7444a93069b411636bdb" exitCode=0 Oct 31 13:33:15.076259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:15.075788 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" event={"ID":"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d","Type":"ContainerDied","Data":"965f8963eb8cb9c73c2014686e459ee16690946c6eed7444a93069b411636bdb"} Oct 31 13:33:15.076432 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:15.076411 5532 scope.go:122] "RemoveContainer" containerID="965f8963eb8cb9c73c2014686e459ee16690946c6eed7444a93069b411636bdb" Oct 31 13:33:15.079472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:15.079437 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:15.079726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:15.079698 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:15.079812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:15.079761 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:16.088511 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:16.088478 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" event={"ID":"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d","Type":"ContainerStarted","Data":"26f047daca6017c7c02df7147ef527c347d7161d4239fcf08f8e23df8b8f9036"} Oct 31 13:33:16.163988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:16.163952 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:33:16.165460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:16.165432 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 13:33:16.165680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:16.165484 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 13:33:17.156618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:17.156447 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 13:33:17.156618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:17.156522 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 13:33:17.643664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:17.642713 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:33:18.156378 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:18.155942 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 13:33:18.156378 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:18.156001 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 13:33:19.158865 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:19.158826 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 13:33:19.160262 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:19.160229 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 13:33:20.157627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:20.155201 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 13:33:20.157627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:20.155264 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 13:33:21.151884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:21.151846 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Oct 31 13:33:21.152376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:21.151905 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Oct 31 13:33:22.156232 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:22.156200 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:33:22.158058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:22.158033 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:33:27.144880 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:27.144842 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68"] Oct 31 13:33:31.780761 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:33:31.780718 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:33:32.145683 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:32.145639 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" podUID="5eb5abfa-b925-49ee-8b96-383bb328b6d0" containerName="payload-processing" containerID="cri-o://1d21ff5f20085f1bddb6ecd3f5cb792947772bb88c68419e6172a7dd7e112da4" gracePeriod=25 Oct 31 13:33:32.265434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:32.265405 5532 generic.go:356] "Generic (PLEG): container finished" podID="5eb5abfa-b925-49ee-8b96-383bb328b6d0" containerID="1d21ff5f20085f1bddb6ecd3f5cb792947772bb88c68419e6172a7dd7e112da4" exitCode=0 Oct 31 13:33:32.265540 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:32.265483 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" event={"ID":"5eb5abfa-b925-49ee-8b96-383bb328b6d0","Type":"ContainerDied","Data":"1d21ff5f20085f1bddb6ecd3f5cb792947772bb88c68419e6172a7dd7e112da4"} Oct 31 13:33:32.435804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:32.435772 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" Oct 31 13:33:32.538148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:32.538117 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb5abfa-b925-49ee-8b96-383bb328b6d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb5abfa-b925-49ee-8b96-383bb328b6d0-plugins-config-volume\") pod \"5eb5abfa-b925-49ee-8b96-383bb328b6d0\" (UID: \"5eb5abfa-b925-49ee-8b96-383bb328b6d0\") " Oct 31 13:33:32.538340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:32.538224 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb5abfa-b925-49ee-8b96-383bb328b6d0-kube-api-access-ddksq\" (UniqueName: \"kubernetes.io/projected/5eb5abfa-b925-49ee-8b96-383bb328b6d0-kube-api-access-ddksq\") pod \"5eb5abfa-b925-49ee-8b96-383bb328b6d0\" (UID: \"5eb5abfa-b925-49ee-8b96-383bb328b6d0\") " Oct 31 13:33:32.539042 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:32.539010 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb5abfa-b925-49ee-8b96-383bb328b6d0-plugins-config-volume" pod "5eb5abfa-b925-49ee-8b96-383bb328b6d0" (UID: "5eb5abfa-b925-49ee-8b96-383bb328b6d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:33:32.541313 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:32.541278 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb5abfa-b925-49ee-8b96-383bb328b6d0-kube-api-access-ddksq" pod "5eb5abfa-b925-49ee-8b96-383bb328b6d0" (UID: "5eb5abfa-b925-49ee-8b96-383bb328b6d0"). InnerVolumeSpecName "kube-api-access-ddksq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:33:32.648713 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:32.641468 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5eb5abfa-b925-49ee-8b96-383bb328b6d0-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:32.648713 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:32.641511 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddksq\" (UniqueName: \"kubernetes.io/projected/5eb5abfa-b925-49ee-8b96-383bb328b6d0-kube-api-access-ddksq\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:33.276816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:33.276791 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" Oct 31 13:33:33.277259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:33.276833 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68" event={"ID":"5eb5abfa-b925-49ee-8b96-383bb328b6d0","Type":"ContainerDied","Data":"116970d5f8cf942cf70b539194f0c5b9c4660b5c8ba97d50cdeda662406166fc"} Oct 31 13:33:33.277259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:33.276899 5532 scope.go:122] "RemoveContainer" containerID="1d21ff5f20085f1bddb6ecd3f5cb792947772bb88c68419e6172a7dd7e112da4" Oct 31 13:33:33.307051 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:33.307022 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68"] Oct 31 13:33:33.311899 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:33.311877 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a8c1600c-7d96b7788c-gxm68"] Oct 31 13:33:34.005667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:34.005637 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb5abfa-b925-49ee-8b96-383bb328b6d0" path="/var/lib/kubelet/pods/5eb5abfa-b925-49ee-8b96-383bb328b6d0/volumes" Oct 31 13:33:35.603493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:35.603455 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw"] Oct 31 13:33:35.604073 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:35.603748 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" podUID="47b9630f-8938-4110-b18b-c7591eaab085" containerName="maas-api" containerID="cri-o://0cc0edede1823b21b1dbae33dbddcb339bea9d38d0a40266cc20c05a350284b6" gracePeriod=30 Oct 31 13:33:36.117199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.117164 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:33:36.177132 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.177091 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-maas-api-tls\") pod \"47b9630f-8938-4110-b18b-c7591eaab085\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " Oct 31 13:33:36.177406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.177389 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47b9630f-8938-4110-b18b-c7591eaab085-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b9630f-8938-4110-b18b-c7591eaab085-openshift-service-ca\") pod \"47b9630f-8938-4110-b18b-c7591eaab085\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " Oct 31 13:33:36.177569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.177551 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47b9630f-8938-4110-b18b-c7591eaab085-kube-api-access-24fjn\" (UniqueName: \"kubernetes.io/projected/47b9630f-8938-4110-b18b-c7591eaab085-kube-api-access-24fjn\") pod \"47b9630f-8938-4110-b18b-c7591eaab085\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " Oct 31 13:33:36.177666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.177650 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-metrics-certs\") pod \"47b9630f-8938-4110-b18b-c7591eaab085\" (UID: \"47b9630f-8938-4110-b18b-c7591eaab085\") " Oct 31 13:33:36.177925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.177894 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47b9630f-8938-4110-b18b-c7591eaab085-openshift-service-ca" pod "47b9630f-8938-4110-b18b-c7591eaab085" (UID: "47b9630f-8938-4110-b18b-c7591eaab085"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:33:36.178386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.178363 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47b9630f-8938-4110-b18b-c7591eaab085-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:36.183432 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.180454 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-metrics-certs" pod "47b9630f-8938-4110-b18b-c7591eaab085" (UID: "47b9630f-8938-4110-b18b-c7591eaab085"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:33:36.183432 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.182357 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-maas-api-tls" pod "47b9630f-8938-4110-b18b-c7591eaab085" (UID: "47b9630f-8938-4110-b18b-c7591eaab085"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:33:36.183750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.183729 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47b9630f-8938-4110-b18b-c7591eaab085-kube-api-access-24fjn" pod "47b9630f-8938-4110-b18b-c7591eaab085" (UID: "47b9630f-8938-4110-b18b-c7591eaab085"). InnerVolumeSpecName "kube-api-access-24fjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:33:36.280548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.280509 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24fjn\" (UniqueName: \"kubernetes.io/projected/47b9630f-8938-4110-b18b-c7591eaab085-kube-api-access-24fjn\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:36.280548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.280547 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:36.280548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.280562 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47b9630f-8938-4110-b18b-c7591eaab085-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:33:36.308083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.308048 5532 generic.go:356] "Generic (PLEG): container finished" podID="47b9630f-8938-4110-b18b-c7591eaab085" containerID="0cc0edede1823b21b1dbae33dbddcb339bea9d38d0a40266cc20c05a350284b6" exitCode=0 Oct 31 13:33:36.308254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.308094 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" Oct 31 13:33:36.308254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.308129 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" event={"ID":"47b9630f-8938-4110-b18b-c7591eaab085","Type":"ContainerDied","Data":"0cc0edede1823b21b1dbae33dbddcb339bea9d38d0a40266cc20c05a350284b6"} Oct 31 13:33:36.308254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.308195 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw" event={"ID":"47b9630f-8938-4110-b18b-c7591eaab085","Type":"ContainerDied","Data":"62f59507b6d2282bee44134536fe626098e98f4c3897a3db8d9b1805d4b84e07"} Oct 31 13:33:36.308254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.308223 5532 scope.go:122] "RemoveContainer" containerID="0cc0edede1823b21b1dbae33dbddcb339bea9d38d0a40266cc20c05a350284b6" Oct 31 13:33:36.320870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.320849 5532 scope.go:122] "RemoveContainer" containerID="0cc0edede1823b21b1dbae33dbddcb339bea9d38d0a40266cc20c05a350284b6" Oct 31 13:33:36.321205 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:33:36.321179 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cc0edede1823b21b1dbae33dbddcb339bea9d38d0a40266cc20c05a350284b6\": container with ID starting with 0cc0edede1823b21b1dbae33dbddcb339bea9d38d0a40266cc20c05a350284b6 not found: ID does not exist" containerID="0cc0edede1823b21b1dbae33dbddcb339bea9d38d0a40266cc20c05a350284b6" Oct 31 13:33:36.321313 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.321210 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cc0edede1823b21b1dbae33dbddcb339bea9d38d0a40266cc20c05a350284b6"} err="failed to get container status \"0cc0edede1823b21b1dbae33dbddcb339bea9d38d0a40266cc20c05a350284b6\": rpc error: code = NotFound desc = could not find container \"0cc0edede1823b21b1dbae33dbddcb339bea9d38d0a40266cc20c05a350284b6\": container with ID starting with 0cc0edede1823b21b1dbae33dbddcb339bea9d38d0a40266cc20c05a350284b6 not found: ID does not exist" Oct 31 13:33:36.336884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.336821 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw"] Oct 31 13:33:36.341407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:36.341380 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a8c1600c-65cc7b6f44-dr7fw"] Oct 31 13:33:38.007553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:38.007500 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47b9630f-8938-4110-b18b-c7591eaab085" path="/var/lib/kubelet/pods/47b9630f-8938-4110-b18b-c7591eaab085/volumes" Oct 31 13:33:39.378204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.378162 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg"] Oct 31 13:33:39.379150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.379125 5532 cpu_manager.go:414] "containerMap: removing container" podUID="47b9630f-8938-4110-b18b-c7591eaab085" containerName="maas-api" Oct 31 13:33:39.379150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.379150 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47b9630f-8938-4110-b18b-c7591eaab085" containerName="maas-api" Oct 31 13:33:39.379315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.379175 5532 cpu_manager.go:414] "containerMap: removing container" podUID="5eb5abfa-b925-49ee-8b96-383bb328b6d0" containerName="payload-processing" Oct 31 13:33:39.379315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.379183 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb5abfa-b925-49ee-8b96-383bb328b6d0" containerName="payload-processing" Oct 31 13:33:39.379457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.379443 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="47b9630f-8938-4110-b18b-c7591eaab085" containerName="maas-api" Oct 31 13:33:39.379517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.379463 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb5abfa-b925-49ee-8b96-383bb328b6d0" containerName="payload-processing" Oct 31 13:33:39.385026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.385004 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.393301 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.393275 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-466002-openshift-default-dockercfg-2dp2t\"" Oct 31 13:33:39.400261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.400232 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg"] Oct 31 13:33:39.435690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.435646 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgsmc\" (UniqueName: \"kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-kube-api-access-sgsmc\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.435889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.435718 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-token\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.435889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.435793 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-socket\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.435889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.435841 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-credential-socket\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.436014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.435984 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-podinfo\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.436093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.436075 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istiod-ca-cert\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.436136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.436119 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-certs\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.436299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.436276 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-data\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.436375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.436326 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-envoy\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.542496 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.542454 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-podinfo\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.542836 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.542807 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istiod-ca-cert\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.543085 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.543060 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-certs\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.543348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.543330 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-data\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.543481 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.543464 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-envoy\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.543784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.543743 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgsmc\" (UniqueName: \"kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-kube-api-access-sgsmc\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.543967 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.543950 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-token\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.544112 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.544091 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-socket\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.545282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.545263 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-credential-socket\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.546107 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.546086 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-credential-socket\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.547344 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.547001 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istiod-ca-cert\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.552523 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.548777 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-data\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.552523 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.549691 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-socket\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.561948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.557668 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-podinfo\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.572704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.562309 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-certs\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.589413 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.589341 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-envoy\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.600391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.597455 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-token\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.600391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.599789 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgsmc\" (UniqueName: \"kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-kube-api-access-sgsmc\") pod \"e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:39.704054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:39.704018 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:40.043822 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:33:40.043791 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83821ff1_96c3_41a0_a3e7_4940a2c1bb76.slice/crio-07eaf07179a26e89e6e856b407a60da9942cc2b4bc6567a9033321e6a48fb824 WatchSource:0}: Error finding container 07eaf07179a26e89e6e856b407a60da9942cc2b4bc6567a9033321e6a48fb824: Status 404 returned error can't find the container with id 07eaf07179a26e89e6e856b407a60da9942cc2b4bc6567a9033321e6a48fb824 Oct 31 13:33:40.044199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:40.044179 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg"] Oct 31 13:33:40.046745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:40.046700 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:40.046863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:40.046782 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:40.046863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:40.046811 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:40.372468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:40.372354 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" event={"ID":"83821ff1-96c3-41a0-a3e7-4940a2c1bb76","Type":"ContainerStarted","Data":"c8417332ecee36c7a2c0bec68c7fb900f3024294fa5735e5ad97b477ab5aa67f"} Oct 31 13:33:40.372468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:40.372466 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" event={"ID":"83821ff1-96c3-41a0-a3e7-4940a2c1bb76","Type":"ContainerStarted","Data":"07eaf07179a26e89e6e856b407a60da9942cc2b4bc6567a9033321e6a48fb824"} Oct 31 13:33:40.402661 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:40.402588 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" podStartSLOduration=1.402572803 podStartE2EDuration="1.402572803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:33:40.400725302 +0000 UTC m=+1788.546469130" watchObservedRunningTime="2026-10-31 13:33:40.402572803 +0000 UTC m=+1788.548316637" Oct 31 13:33:40.705156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:40.705117 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:41.711504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:41.711473 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:42.388837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:42.388805 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:42.389680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:42.389661 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:33:44.116408 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:44.116368 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz"] Oct 31 13:33:44.124905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:44.124875 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" Oct 31 13:33:44.139227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:44.139117 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-466002-dockercfg-xh64n\"" Oct 31 13:33:44.139227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:44.139186 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-466002\"" Oct 31 13:33:44.143670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:44.143642 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz"] Oct 31 13:33:44.272049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:44.272010 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjlx2\" (UniqueName: \"kubernetes.io/projected/21375def-9f25-4f90-a449-2bdff0362119-kube-api-access-qjlx2\") pod \"payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz\" (UID: \"21375def-9f25-4f90-a449-2bdff0362119\") " pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" Oct 31 13:33:44.272320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:44.272069 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21375def-9f25-4f90-a449-2bdff0362119-plugins-config-volume\") pod \"payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz\" (UID: \"21375def-9f25-4f90-a449-2bdff0362119\") " pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" Oct 31 13:33:44.376670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:44.373900 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjlx2\" (UniqueName: \"kubernetes.io/projected/21375def-9f25-4f90-a449-2bdff0362119-kube-api-access-qjlx2\") pod \"payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz\" (UID: \"21375def-9f25-4f90-a449-2bdff0362119\") " pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" Oct 31 13:33:44.376670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:44.373984 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21375def-9f25-4f90-a449-2bdff0362119-plugins-config-volume\") pod \"payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz\" (UID: \"21375def-9f25-4f90-a449-2bdff0362119\") " pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" Oct 31 13:33:44.376670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:44.375140 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21375def-9f25-4f90-a449-2bdff0362119-plugins-config-volume\") pod \"payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz\" (UID: \"21375def-9f25-4f90-a449-2bdff0362119\") " pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" Oct 31 13:33:44.429726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:44.429082 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjlx2\" (UniqueName: \"kubernetes.io/projected/21375def-9f25-4f90-a449-2bdff0362119-kube-api-access-qjlx2\") pod \"payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz\" (UID: \"21375def-9f25-4f90-a449-2bdff0362119\") " pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" Oct 31 13:33:44.465615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:44.461110 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" Oct 31 13:33:44.852112 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:44.852087 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz"] Oct 31 13:33:44.853265 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:33:44.853238 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21375def_9f25_4f90_a449_2bdff0362119.slice/crio-22fe651ea4dfd08fcffdeddd06a03ed9397521668d826fa21e76059535d1e05d WatchSource:0}: Error finding container 22fe651ea4dfd08fcffdeddd06a03ed9397521668d826fa21e76059535d1e05d: Status 404 returned error can't find the container with id 22fe651ea4dfd08fcffdeddd06a03ed9397521668d826fa21e76059535d1e05d Oct 31 13:33:45.044890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:45.044855 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8"] Oct 31 13:33:45.049267 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:45.049243 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a172f8a-r2nv8"] Oct 31 13:33:45.464663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:45.464627 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" event={"ID":"21375def-9f25-4f90-a449-2bdff0362119","Type":"ContainerStarted","Data":"563ae7f49f8ff8a499259a3e9fdecb5cb8d3fc4f7dfecf9297cc7f243b3bdd46"} Oct 31 13:33:45.465062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:45.464690 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" event={"ID":"21375def-9f25-4f90-a449-2bdff0362119","Type":"ContainerStarted","Data":"22fe651ea4dfd08fcffdeddd06a03ed9397521668d826fa21e76059535d1e05d"} Oct 31 13:33:45.465062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:45.464759 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" Oct 31 13:33:45.495286 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:45.495229 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" podStartSLOduration=1.495213106 podStartE2EDuration="1.495213106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:33:45.489321945 +0000 UTC m=+1793.635065773" watchObservedRunningTime="2026-10-31 13:33:45.495213106 +0000 UTC m=+1793.640956933" Oct 31 13:33:46.006422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:46.006389 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0916e8e9-3cae-4e5b-86db-ea508f3d24f4" path="/var/lib/kubelet/pods/0916e8e9-3cae-4e5b-86db-ea508f3d24f4/volumes" Oct 31 13:33:49.443280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:49.443197 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f"] Oct 31 13:33:49.449292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:49.449263 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" Oct 31 13:33:49.479702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:49.477082 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-33c10a-dockercfg-6w6st\"" Oct 31 13:33:49.479702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:49.477171 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-33c10a\"" Oct 31 13:33:49.542702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:49.531675 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c9c6\" (UniqueName: \"kubernetes.io/projected/de6b3aca-3802-48c4-9315-2492f01492a0-kube-api-access-4c9c6\") pod \"payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f\" (UID: \"de6b3aca-3802-48c4-9315-2492f01492a0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" Oct 31 13:33:49.591843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:49.575943 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6b3aca-3802-48c4-9315-2492f01492a0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f\" (UID: \"de6b3aca-3802-48c4-9315-2492f01492a0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" Oct 31 13:33:49.591843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:49.564551 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f"] Oct 31 13:33:49.685824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:49.685776 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6b3aca-3802-48c4-9315-2492f01492a0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f\" (UID: \"de6b3aca-3802-48c4-9315-2492f01492a0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" Oct 31 13:33:49.687740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:49.687692 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c9c6\" (UniqueName: \"kubernetes.io/projected/de6b3aca-3802-48c4-9315-2492f01492a0-kube-api-access-4c9c6\") pod \"payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f\" (UID: \"de6b3aca-3802-48c4-9315-2492f01492a0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" Oct 31 13:33:49.689518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:49.689470 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6b3aca-3802-48c4-9315-2492f01492a0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f\" (UID: \"de6b3aca-3802-48c4-9315-2492f01492a0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" Oct 31 13:33:49.720788 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:49.720706 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c9c6\" (UniqueName: \"kubernetes.io/projected/de6b3aca-3802-48c4-9315-2492f01492a0-kube-api-access-4c9c6\") pod \"payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f\" (UID: \"de6b3aca-3802-48c4-9315-2492f01492a0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" Oct 31 13:33:49.831969 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:49.831935 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" Oct 31 13:33:50.150437 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:33:50.150408 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6b3aca_3802_48c4_9315_2492f01492a0.slice/crio-506215e8a6e912a860a5eacbe37372326739da6acd61c71019c682e9db0a6b6e WatchSource:0}: Error finding container 506215e8a6e912a860a5eacbe37372326739da6acd61c71019c682e9db0a6b6e: Status 404 returned error can't find the container with id 506215e8a6e912a860a5eacbe37372326739da6acd61c71019c682e9db0a6b6e Oct 31 13:33:50.150589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:50.150556 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f"] Oct 31 13:33:50.686210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:50.686159 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" event={"ID":"de6b3aca-3802-48c4-9315-2492f01492a0","Type":"ContainerStarted","Data":"7ba608848fc44cfe24b5895340fb2dacd05f9085ac4ab9130847d3cd6fff3511"} Oct 31 13:33:50.686746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:50.686239 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" event={"ID":"de6b3aca-3802-48c4-9315-2492f01492a0","Type":"ContainerStarted","Data":"506215e8a6e912a860a5eacbe37372326739da6acd61c71019c682e9db0a6b6e"} Oct 31 13:33:50.686746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:50.686354 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" Oct 31 13:33:50.714961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:50.714912 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" podStartSLOduration=1.7148984139999999 podStartE2EDuration="1.714898414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:33:50.713462571 +0000 UTC m=+1798.859206410" watchObservedRunningTime="2026-10-31 13:33:50.714898414 +0000 UTC m=+1798.860642239" Oct 31 13:33:52.415921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:52.415889 5532 scope.go:122] "RemoveContainer" containerID="73f4b6d65df5bdbe79bb3ecd33305e7efa04cd0c8809da9abe3a7aa38c3339d5" Oct 31 13:33:52.430226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:52.430201 5532 scope.go:122] "RemoveContainer" containerID="906bb7b5e191ea65ce5731aff8a60ac18155789b25ba1f7dd5f1147cef80177e" Oct 31 13:33:52.440962 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:52.440939 5532 scope.go:122] "RemoveContainer" containerID="0f41719e68eb2c25a6a14568dc35914b83041679f6c1f66b6131d392c9bd3d2b" Oct 31 13:33:52.451394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:52.451374 5532 scope.go:122] "RemoveContainer" containerID="c0ba86706adfcc24dbcc1695a1e50260153ecee2d65f108dd30ad5dfcd6bb73e" Oct 31 13:33:52.461856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:52.461833 5532 scope.go:122] "RemoveContainer" containerID="e25864b0c149da5f20659da4046e28fde573bb4eb2bd50232bc8c633fcd10a91" Oct 31 13:33:52.471948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:52.471927 5532 scope.go:122] "RemoveContainer" containerID="b7bc34469af3e2d6073870052ca332a6ae1a2dc3222e1dd840f3e94736ef491c" Oct 31 13:33:52.481519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:52.481504 5532 scope.go:122] "RemoveContainer" containerID="082c00227f8d21cf24acbf4eecc792efdeeaf73328fa42a62c8eccd177bf5e13" Oct 31 13:33:52.490959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:52.490941 5532 scope.go:122] "RemoveContainer" containerID="399039575ab60a47e385a416438be0150834fcaa3f66e2b19d14fa2bf934937b" Oct 31 13:33:52.500127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:52.500097 5532 scope.go:122] "RemoveContainer" containerID="4e24ed62464b25ff868db7295360943e051b3ba651fb4920f730a33d30d7cfcc" Oct 31 13:33:52.509312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:52.509287 5532 scope.go:122] "RemoveContainer" containerID="13262a247314c8ee33f6ae6f789d1535df87a5fdf328cba42e1b7243c9689e79" Oct 31 13:33:55.669664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.669627 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps"] Oct 31 13:33:55.677577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.677552 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.687117 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.687088 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-75c956-openshift-default-dockercfg-qxrpj\"" Oct 31 13:33:55.697425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.697400 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps"] Oct 31 13:33:55.711787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.711756 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-envoy\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.711923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.711843 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-podinfo\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.711998 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.711980 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v7zx\" (UniqueName: \"kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-kube-api-access-4v7zx\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.712044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.712035 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-socket\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.712093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.712062 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-data\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.712195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.712170 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-token\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.712335 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.712315 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-credential-socket\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.712412 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.712395 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb475d1-1432-4095-b9c1-e6b16f18079a-istiod-ca-cert\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.712458 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.712443 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-certs\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.816654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.814112 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-podinfo\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.816654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.814254 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v7zx\" (UniqueName: \"kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-kube-api-access-4v7zx\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.816654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.814305 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-socket\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.816654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.814330 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-data\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.816654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.814362 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-token\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.816654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.815707 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-credential-socket\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.816654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.815781 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb475d1-1432-4095-b9c1-e6b16f18079a-istiod-ca-cert\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.816654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.815832 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-certs\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.816654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.816218 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-envoy\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.816654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.816297 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-socket\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.839651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.817378 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb475d1-1432-4095-b9c1-e6b16f18079a-istiod-ca-cert\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.839651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.818008 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-credential-socket\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.839651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.818269 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-certs\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.839651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.829847 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-data\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.839651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.834526 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-envoy\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.861616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.842432 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-podinfo\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.877612 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.875206 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-token\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.905762 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.889121 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v7zx\" (UniqueName: \"kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-kube-api-access-4v7zx\") pod \"e2e-api-a-75c956-openshift-default-6995b9c56c-hckps\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:55.995096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:55.995008 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:56.032518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:56.032488 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc"] Oct 31 13:33:56.037259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:56.037235 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40efb871-jl4zc"] Oct 31 13:33:56.346451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:56.346424 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps"] Oct 31 13:33:56.347935 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:33:56.347904 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceb475d1_1432_4095_b9c1_e6b16f18079a.slice/crio-40b9b385cf6b90b9481f420dae11527da8da8c3328a05ddb12f1e1324749e3da WatchSource:0}: Error finding container 40b9b385cf6b90b9481f420dae11527da8da8c3328a05ddb12f1e1324749e3da: Status 404 returned error can't find the container with id 40b9b385cf6b90b9481f420dae11527da8da8c3328a05ddb12f1e1324749e3da Oct 31 13:33:56.351769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:56.351499 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:56.351769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:56.351630 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:56.351769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:56.351677 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:33:56.475665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:56.475586 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" Oct 31 13:33:56.752952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:56.752910 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" event={"ID":"ceb475d1-1432-4095-b9c1-e6b16f18079a","Type":"ContainerStarted","Data":"a92b1937e5b263f981c46c09a21fc0f1fc390cbcfac942d4ca2f317a02979f4a"} Oct 31 13:33:56.753419 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:56.752989 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" event={"ID":"ceb475d1-1432-4095-b9c1-e6b16f18079a","Type":"ContainerStarted","Data":"40b9b385cf6b90b9481f420dae11527da8da8c3328a05ddb12f1e1324749e3da"} Oct 31 13:33:56.782023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:56.781972 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" podStartSLOduration=1.781957572 podStartE2EDuration="1.781957572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:33:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:33:56.777144497 +0000 UTC m=+1804.922888327" watchObservedRunningTime="2026-10-31 13:33:56.781957572 +0000 UTC m=+1804.927701400" Oct 31 13:33:56.995417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:56.995385 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:58.038316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:58.038274 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b81d889-d246-4041-8380-4974723b2689" path="/var/lib/kubelet/pods/7b81d889-d246-4041-8380-4974723b2689/volumes" Oct 31 13:33:58.039069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:58.038905 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:58.769569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:58.769536 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:58.770503 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:58.770478 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:33:59.353770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:33:59.353736 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:34:00.470989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.470906 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w"] Oct 31 13:34:00.477352 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.477324 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" Oct 31 13:34:00.489686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.489655 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-75c956\"" Oct 31 13:34:00.495791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.495761 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w"] Oct 31 13:34:00.501079 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.501046 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82"] Oct 31 13:34:00.523617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.512457 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-75c956-dockercfg-cms67\"" Oct 31 13:34:00.528234 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.524017 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" Oct 31 13:34:00.534893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.534869 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82"] Oct 31 13:34:00.644633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.641432 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb9gj\" (UniqueName: \"kubernetes.io/projected/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-kube-api-access-wb9gj\") pod \"payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82\" (UID: \"76cc343f-d6c3-4df5-95b4-4384bd95e2d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" Oct 31 13:34:00.672617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.641670 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82\" (UID: \"76cc343f-d6c3-4df5-95b4-4384bd95e2d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" Oct 31 13:34:00.672617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.661371 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f031a89b-1620-4cf6-a62c-4d73bce14623-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w\" (UID: \"f031a89b-1620-4cf6-a62c-4d73bce14623\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" Oct 31 13:34:00.672617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.661677 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktcw\" (UniqueName: \"kubernetes.io/projected/f031a89b-1620-4cf6-a62c-4d73bce14623-kube-api-access-pktcw\") pod \"payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w\" (UID: \"f031a89b-1620-4cf6-a62c-4d73bce14623\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" Oct 31 13:34:00.763566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.763470 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktcw\" (UniqueName: \"kubernetes.io/projected/f031a89b-1620-4cf6-a62c-4d73bce14623-kube-api-access-pktcw\") pod \"payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w\" (UID: \"f031a89b-1620-4cf6-a62c-4d73bce14623\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" Oct 31 13:34:00.763749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.763588 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb9gj\" (UniqueName: \"kubernetes.io/projected/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-kube-api-access-wb9gj\") pod \"payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82\" (UID: \"76cc343f-d6c3-4df5-95b4-4384bd95e2d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" Oct 31 13:34:00.763749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.763670 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82\" (UID: \"76cc343f-d6c3-4df5-95b4-4384bd95e2d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" Oct 31 13:34:00.763856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.763781 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f031a89b-1620-4cf6-a62c-4d73bce14623-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w\" (UID: \"f031a89b-1620-4cf6-a62c-4d73bce14623\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" Oct 31 13:34:00.764721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.764689 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f031a89b-1620-4cf6-a62c-4d73bce14623-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w\" (UID: \"f031a89b-1620-4cf6-a62c-4d73bce14623\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" Oct 31 13:34:00.764961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.764938 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-plugins-config-volume\") pod \"payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82\" (UID: \"76cc343f-d6c3-4df5-95b4-4384bd95e2d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" Oct 31 13:34:00.777283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.777255 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktcw\" (UniqueName: \"kubernetes.io/projected/f031a89b-1620-4cf6-a62c-4d73bce14623-kube-api-access-pktcw\") pod \"payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w\" (UID: \"f031a89b-1620-4cf6-a62c-4d73bce14623\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" Oct 31 13:34:00.777401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.777313 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb9gj\" (UniqueName: \"kubernetes.io/projected/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-kube-api-access-wb9gj\") pod \"payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82\" (UID: \"76cc343f-d6c3-4df5-95b4-4384bd95e2d8\") " pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" Oct 31 13:34:00.827158 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.827120 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" Oct 31 13:34:00.852900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:00.852867 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" Oct 31 13:34:01.431067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:01.430920 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w"] Oct 31 13:34:01.434094 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:34:01.434068 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cc343f_d6c3_4df5_95b4_4384bd95e2d8.slice/crio-3b24534c01822925791c0bf67620df2c158ecdc1c62a46adee344222635d7e36 WatchSource:0}: Error finding container 3b24534c01822925791c0bf67620df2c158ecdc1c62a46adee344222635d7e36: Status 404 returned error can't find the container with id 3b24534c01822925791c0bf67620df2c158ecdc1c62a46adee344222635d7e36 Oct 31 13:34:01.442388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:01.442356 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82"] Oct 31 13:34:01.720751 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:01.720719 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" Oct 31 13:34:01.819043 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:01.819008 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" event={"ID":"76cc343f-d6c3-4df5-95b4-4384bd95e2d8","Type":"ContainerStarted","Data":"3b24534c01822925791c0bf67620df2c158ecdc1c62a46adee344222635d7e36"} Oct 31 13:34:01.821478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:01.821447 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" event={"ID":"f031a89b-1620-4cf6-a62c-4d73bce14623","Type":"ContainerStarted","Data":"4a77625a66fdfe528e28bbaffe9b9b958a9c1f60fc7a56772cca2f802686cc45"} Oct 31 13:34:01.822044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:01.822026 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" Oct 31 13:34:01.822207 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:34:01.821380 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:34:01.849418 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:01.848168 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" podStartSLOduration=1.8481493310000001 podStartE2EDuration="1.848149331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:01.847009832 +0000 UTC m=+1809.992753662" watchObservedRunningTime="2026-10-31 13:34:01.848149331 +0000 UTC m=+1809.993893160" Oct 31 13:34:02.831943 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:02.831907 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" event={"ID":"76cc343f-d6c3-4df5-95b4-4384bd95e2d8","Type":"ContainerStarted","Data":"aec8ddd7a42607a5eafe34137df9d20f929129d2f1860669fb2a77362933bd35"} Oct 31 13:34:02.832418 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:02.831981 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" Oct 31 13:34:02.833783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:02.833753 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" event={"ID":"f031a89b-1620-4cf6-a62c-4d73bce14623","Type":"ContainerStarted","Data":"211dfa39dcec5b0175f589f82e753f4087d87a6b953d43ededa5e7ab6fe73915"} Oct 31 13:34:02.860432 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:02.860384 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" podStartSLOduration=2.8603693 podStartE2EDuration="2.8603693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:02.856062548 +0000 UTC m=+1811.001806378" watchObservedRunningTime="2026-10-31 13:34:02.8603693 +0000 UTC m=+1811.006113127" Oct 31 13:34:10.039877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:10.039840 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5"] Oct 31 13:34:10.044320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:10.044296 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e4e5df1-s8wt5"] Oct 31 13:34:12.005853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:12.005814 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d9b75a-ffdd-452e-a13f-acb001009bf4" path="/var/lib/kubelet/pods/e4d9b75a-ffdd-452e-a13f-acb001009bf4/volumes" Oct 31 13:34:13.844753 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:13.844724 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" Oct 31 13:34:13.845148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:13.844801 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" Oct 31 13:34:18.034241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:18.034208 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8"] Oct 31 13:34:18.039967 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:18.039940 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1ec2677-nb6b8"] Oct 31 13:34:20.007147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:20.007107 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93043d3-9e55-4d63-85e7-47e7e04076c0" path="/var/lib/kubelet/pods/f93043d3-9e55-4d63-85e7-47e7e04076c0/volumes" Oct 31 13:34:24.888077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:24.888030 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9"] Oct 31 13:34:24.894763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:24.894735 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" Oct 31 13:34:24.904289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:24.904238 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-abdcb1-dockercfg-d8qwm\"" Oct 31 13:34:24.904581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:24.904556 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-abdcb1\"" Oct 31 13:34:24.910525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:24.910498 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9"] Oct 31 13:34:25.012620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:25.003818 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2c77\" (UniqueName: \"kubernetes.io/projected/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-kube-api-access-l2c77\") pod \"payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9\" (UID: \"a5ae713b-d2f7-45c0-bf5c-166d580bc32d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" Oct 31 13:34:25.039833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:25.039791 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9\" (UID: \"a5ae713b-d2f7-45c0-bf5c-166d580bc32d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" Oct 31 13:34:25.200137 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:25.187833 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2c77\" (UniqueName: \"kubernetes.io/projected/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-kube-api-access-l2c77\") pod \"payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9\" (UID: \"a5ae713b-d2f7-45c0-bf5c-166d580bc32d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" Oct 31 13:34:25.200137 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:25.198552 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9\" (UID: \"a5ae713b-d2f7-45c0-bf5c-166d580bc32d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" Oct 31 13:34:25.200137 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:25.199855 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9\" (UID: \"a5ae713b-d2f7-45c0-bf5c-166d580bc32d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" Oct 31 13:34:25.241171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:25.234416 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2c77\" (UniqueName: \"kubernetes.io/projected/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-kube-api-access-l2c77\") pod \"payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9\" (UID: \"a5ae713b-d2f7-45c0-bf5c-166d580bc32d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" Oct 31 13:34:25.518718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:25.518621 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" Oct 31 13:34:25.961404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:25.961370 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9"] Oct 31 13:34:25.961879 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:34:25.961440 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5ae713b_d2f7_45c0_bf5c_166d580bc32d.slice/crio-50b2e4a7dba614e089a57de2da9a06e0065884f029508da63a063f8ab8bfd7e2 WatchSource:0}: Error finding container 50b2e4a7dba614e089a57de2da9a06e0065884f029508da63a063f8ab8bfd7e2: Status 404 returned error can't find the container with id 50b2e4a7dba614e089a57de2da9a06e0065884f029508da63a063f8ab8bfd7e2 Oct 31 13:34:26.184050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:26.184011 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt"] Oct 31 13:34:26.197586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:26.197553 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4"] Oct 31 13:34:26.226138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:26.225959 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" event={"ID":"a5ae713b-d2f7-45c0-bf5c-166d580bc32d","Type":"ContainerStarted","Data":"f3c0116373c509c978317c2fd17a17233185cd8f4391ba0030df70e8368bbc25"} Oct 31 13:34:26.226138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:26.226052 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" event={"ID":"a5ae713b-d2f7-45c0-bf5c-166d580bc32d","Type":"ContainerStarted","Data":"50b2e4a7dba614e089a57de2da9a06e0065884f029508da63a063f8ab8bfd7e2"} Oct 31 13:34:26.226697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:26.226674 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" Oct 31 13:34:26.262805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:26.262727 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" podStartSLOduration=2.262708318 podStartE2EDuration="2.262708318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:26.258265483 +0000 UTC m=+1834.404009308" watchObservedRunningTime="2026-10-31 13:34:26.262708318 +0000 UTC m=+1834.408452147" Oct 31 13:34:31.185182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:31.185132 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" podUID="7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2" containerName="payload-processing" containerID="cri-o://b32b3ab39d686004880790adf10c37dda14d770f2dd40e659e970b36d566ae50" gracePeriod=25 Oct 31 13:34:31.198897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:31.198823 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" podUID="ededb5f2-22bb-4d1f-896a-63c8cc779422" containerName="payload-pre-processing" containerID="cri-o://d2b7d41ae02f7da641319599873650ee5a64ea18a9531ce5b3b6747c305b9405" gracePeriod=25 Oct 31 13:34:31.764105 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:34:31.764065 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:34:36.287679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:36.287634 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs"] Oct 31 13:34:36.387926 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:36.387892 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs"] Oct 31 13:34:36.388403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:36.388381 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" Oct 31 13:34:36.431211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:36.407892 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cce3e7-dockercfg-ncffd\"" Oct 31 13:34:36.490619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:36.489839 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51aea716-c142-4909-a102-5f46f1c68991-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs\" (UID: \"51aea716-c142-4909-a102-5f46f1c68991\") " pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" Oct 31 13:34:36.490619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:36.490090 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v26kf\" (UniqueName: \"kubernetes.io/projected/51aea716-c142-4909-a102-5f46f1c68991-kube-api-access-v26kf\") pod \"payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs\" (UID: \"51aea716-c142-4909-a102-5f46f1c68991\") " pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" Oct 31 13:34:36.498257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:36.498131 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cce3e7\"" Oct 31 13:34:36.604641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:36.597430 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v26kf\" (UniqueName: \"kubernetes.io/projected/51aea716-c142-4909-a102-5f46f1c68991-kube-api-access-v26kf\") pod \"payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs\" (UID: \"51aea716-c142-4909-a102-5f46f1c68991\") " pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" Oct 31 13:34:36.622983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:36.622830 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51aea716-c142-4909-a102-5f46f1c68991-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs\" (UID: \"51aea716-c142-4909-a102-5f46f1c68991\") " pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" Oct 31 13:34:36.641619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:36.625222 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51aea716-c142-4909-a102-5f46f1c68991-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs\" (UID: \"51aea716-c142-4909-a102-5f46f1c68991\") " pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" Oct 31 13:34:36.752970 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:36.752140 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v26kf\" (UniqueName: \"kubernetes.io/projected/51aea716-c142-4909-a102-5f46f1c68991-kube-api-access-v26kf\") pod \"payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs\" (UID: \"51aea716-c142-4909-a102-5f46f1c68991\") " pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" Oct 31 13:34:36.919945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:36.919912 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" Oct 31 13:34:37.318243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:37.318145 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" Oct 31 13:34:37.622719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:37.586270 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:37.622719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:37.587371 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:37.622719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:37.601737 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:34:37.715717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:37.715671 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:38.023576 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.021702 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:38.233624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.192382 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:38.238169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.238135 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:38.330618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.259882 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:38.362619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.279341 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:34:38.362619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.279396 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:38.362619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.279415 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:34:38.392855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.378136 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs"] Oct 31 13:34:38.420889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.416818 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:38.608033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.607933 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:34:38.632619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.609758 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:34:38.668093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.616116 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:34:38.668709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.668680 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:34:38.842766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.823697 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" podUID="7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:34:38.939629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:38.927690 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" podUID="ededb5f2-22bb-4d1f-896a-63c8cc779422" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:34:39.099631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.056316 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:39.099631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.096787 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:34:39.327755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.289688 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:39.327755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.290375 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:34:39.327755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.319250 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:39.327755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.319836 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:34:39.360667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.329398 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:39.360667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.336545 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:34:39.360667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.336607 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" event={"ID":"51aea716-c142-4909-a102-5f46f1c68991","Type":"ContainerStarted","Data":"7a84c633b6422d35bbc48e11f73e1a364a379c11458b26e985f6eef510b4ff21"} Oct 31 13:34:39.443302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.363043 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:39.443302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.364217 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:34:39.443302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.365323 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:34:39.443302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.389979 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:34:39.443302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.390048 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:34:39.443302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.390086 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:34:39.564647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.554750 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:39.592534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.592447 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr"] Oct 31 13:34:39.636033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.630241 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:39.716620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.689067 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:39.793619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.792889 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr"] Oct 31 13:34:39.793619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.793014 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:39.804705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.803152 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-abdcb1-serving-cert\"" Oct 31 13:34:39.851787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.812880 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l"] Oct 31 13:34:39.851787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.812937 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c41b41e-w5z6l"] Oct 31 13:34:39.851787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.826971 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:39.851787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.827028 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjhsg\" (UniqueName: \"kubernetes.io/projected/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-kube-api-access-gjhsg\") pod \"maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:39.851787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.827109 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:39.851787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.827202 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:39.853704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.852391 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:34:39.866735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.863370 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:39.910139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.905108 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:34:39.947174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.947130 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:39.970619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.947391 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjhsg\" (UniqueName: \"kubernetes.io/projected/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-kube-api-access-gjhsg\") pod \"maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:39.982640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.971075 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:39.983022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:39.982996 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:40.152678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.048147 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:40.152678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.073152 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:40.152678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.116935 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:40.154438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.154211 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:40.288668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.288046 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce" path="/var/lib/kubelet/pods/e38b6b3c-9451-4da4-b6f6-cbcd2dbd4cce/volumes" Oct 31 13:34:40.303751 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.291670 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:40.489620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.485721 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjhsg\" (UniqueName: \"kubernetes.io/projected/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-kube-api-access-gjhsg\") pod \"maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:40.531629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.526869 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:34:40.531629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.527709 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:40.531629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.529022 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:34:40.531629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.529668 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:34:40.689725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.647795 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:40.690083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.690013 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:40.801619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.801072 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:40.816815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.816780 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:34:40.902017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:40.901974 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:41.086941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.086848 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:41.211620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.161166 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:41.216996 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.212199 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Oct 31 13:34:41.338855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.322075 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:41.503618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.486855 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" podUID="3f952465-49ad-43cf-9598-dd3741e8de2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:41.536618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.528750 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:34:41.627034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.572683 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:41.676650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.666719 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:34:41.779169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.700278 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:41.779169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.700471 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:41.779169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.750647 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:41.820653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.531764 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:41.957185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.889099 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:34:41.957185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.890405 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 13:34:42.011652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.986462 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" event={"ID":"51aea716-c142-4909-a102-5f46f1c68991","Type":"ContainerStarted","Data":"c954889247eb06fdef42e59d0cc0f046a16486f992d1e367131a8739ebcc81d0"} Oct 31 13:34:42.011652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:41.986936 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" Oct 31 13:34:42.112256 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:42.112213 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:34:42.279458 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:42.212667 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-48lfw" podUID="51b75119-a5cd-4e8c-bd06-3725a6bb30a5" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.132.0.29:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:42.279458 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:42.217392 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:42.481771 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:42.346269 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:42.981774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:42.898009 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:43.116545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:42.936435 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:43.123809 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.123764 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:34:43.124135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.124099 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:34:43.124321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.124304 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:34:43.222573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.222526 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:43.244724 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:34:43.232268 5532 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.234s" Oct 31 13:34:43.244724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.237881 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:34:43.244724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.238020 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:34:43.244724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.238059 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:34:43.244724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.238095 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:34:43.294485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.294443 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:34:43.295424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.295391 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:43.357618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.348040 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:43.426845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.421021 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-h4g6w" podUID="3f952465-49ad-43cf-9598-dd3741e8de2c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:34:43.432961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.432937 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:34:43.504309 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.504281 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:34:43.508773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.508736 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:34:43.510702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.510675 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:34:43.510886 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.510865 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:34:43.511316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.511299 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:34:43.512079 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.512029 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" podStartSLOduration=8.512011298000001 podStartE2EDuration="8.512011298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:42.040262591 +0000 UTC m=+1850.186006425" watchObservedRunningTime="2026-10-31 13:34:43.512011298 +0000 UTC m=+1851.657755126" Oct 31 13:34:43.512882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.512861 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:34:43.514424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.514404 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:34:43.519864 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:43.519844 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr"] Oct 31 13:34:44.534323 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:44.534277 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" event={"ID":"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0","Type":"ContainerStarted","Data":"5808c72f31cb540c90cd52cbb911753da1115e288c01e4ed1fbfa26c636fe585"} Oct 31 13:34:44.534836 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:44.534332 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" event={"ID":"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0","Type":"ContainerStarted","Data":"f21ce1afe80cb44ee6f8e7180f7d2c7b8b19ef57efc9ca5982a1f78cdbf400a8"} Oct 31 13:34:44.534929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:44.534909 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:44.557969 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:44.557911 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" podStartSLOduration=6.059884093 podStartE2EDuration="6.557894925s" totalImagesPullingTime="498.010832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:44.556466062 +0000 UTC m=+1852.702209891" watchObservedRunningTime="2026-10-31 13:34:44.557894925 +0000 UTC m=+1852.703638789" Oct 31 13:34:48.764715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:48.764677 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" podUID="7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:34:49.526421 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:48.765003 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" podUID="ededb5f2-22bb-4d1f-896a-63c8cc779422" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:34:49.526421 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:49.036981 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf"] Oct 31 13:34:49.526421 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:49.050495 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-883db104-qhbxf"] Oct 31 13:34:50.006643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:50.006610 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2b78bfc-ece6-422f-838b-f6c8c7f2425a" path="/var/lib/kubelet/pods/c2b78bfc-ece6-422f-838b-f6c8c7f2425a/volumes" Oct 31 13:34:50.649979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:50.649946 5532 generic.go:356] "Generic (PLEG): container finished" podID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerID="a92b1937e5b263f981c46c09a21fc0f1fc390cbcfac942d4ca2f317a02979f4a" exitCode=0 Oct 31 13:34:50.650376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:50.650016 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" event={"ID":"ceb475d1-1432-4095-b9c1-e6b16f18079a","Type":"ContainerDied","Data":"a92b1937e5b263f981c46c09a21fc0f1fc390cbcfac942d4ca2f317a02979f4a"} Oct 31 13:34:50.650499 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:50.650474 5532 scope.go:122] "RemoveContainer" containerID="a92b1937e5b263f981c46c09a21fc0f1fc390cbcfac942d4ca2f317a02979f4a" Oct 31 13:34:50.652554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:50.652523 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:34:50.652738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:50.652654 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:34:50.652738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:50.652704 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:34:50.652857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:50.652781 5532 generic.go:356] "Generic (PLEG): container finished" podID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerID="c8417332ecee36c7a2c0bec68c7fb900f3024294fa5735e5ad97b477ab5aa67f" exitCode=0 Oct 31 13:34:50.652857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:50.652815 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" event={"ID":"83821ff1-96c3-41a0-a3e7-4940a2c1bb76","Type":"ContainerDied","Data":"c8417332ecee36c7a2c0bec68c7fb900f3024294fa5735e5ad97b477ab5aa67f"} Oct 31 13:34:50.653312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:50.653226 5532 scope.go:122] "RemoveContainer" containerID="c8417332ecee36c7a2c0bec68c7fb900f3024294fa5735e5ad97b477ab5aa67f" Oct 31 13:34:50.666391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:50.665673 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:34:50.666391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:50.665922 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:34:50.666391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:50.666019 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:34:51.495932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.495900 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" Oct 31 13:34:51.545285 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.545254 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" Oct 31 13:34:51.600619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.588350 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 13:34:51.633266 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.633234 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-kube-api-access-hrrvk\" (UniqueName: \"kubernetes.io/projected/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-kube-api-access-hrrvk\") pod \"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2\" (UID: \"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2\") " Oct 31 13:34:51.634275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.634247 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-plugins-config-volume\") pod \"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2\" (UID: \"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2\") " Oct 31 13:34:51.634954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.634861 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-plugins-config-volume" pod "7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2" (UID: "7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:34:51.635694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.635646 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ededb5f2-22bb-4d1f-896a-63c8cc779422-plugins-config-volume" pod "ededb5f2-22bb-4d1f-896a-63c8cc779422" (UID: "ededb5f2-22bb-4d1f-896a-63c8cc779422"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:34:51.635834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.635818 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ededb5f2-22bb-4d1f-896a-63c8cc779422-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ededb5f2-22bb-4d1f-896a-63c8cc779422-plugins-config-volume\") pod \"ededb5f2-22bb-4d1f-896a-63c8cc779422\" (UID: \"ededb5f2-22bb-4d1f-896a-63c8cc779422\") " Oct 31 13:34:51.636510 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.636491 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ededb5f2-22bb-4d1f-896a-63c8cc779422-kube-api-access-hh84x\" (UniqueName: \"kubernetes.io/projected/ededb5f2-22bb-4d1f-896a-63c8cc779422-kube-api-access-hh84x\") pod \"ededb5f2-22bb-4d1f-896a-63c8cc779422\" (UID: \"ededb5f2-22bb-4d1f-896a-63c8cc779422\") " Oct 31 13:34:51.637796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.637775 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ededb5f2-22bb-4d1f-896a-63c8cc779422-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:34:51.637925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.637911 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:34:51.675637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.659627 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-kube-api-access-hrrvk" pod "7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2" (UID: "7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2"). InnerVolumeSpecName "kube-api-access-hrrvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:34:51.682629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.679999 5532 generic.go:356] "Generic (PLEG): container finished" podID="7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2" containerID="b32b3ab39d686004880790adf10c37dda14d770f2dd40e659e970b36d566ae50" exitCode=0 Oct 31 13:34:51.682629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.680176 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" event={"ID":"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2","Type":"ContainerDied","Data":"b32b3ab39d686004880790adf10c37dda14d770f2dd40e659e970b36d566ae50"} Oct 31 13:34:51.682629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.680229 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" event={"ID":"7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2","Type":"ContainerDied","Data":"803b288e8cafd9f4f5f55c2c41e48550d04450e10699c5038c5b262f40782a0c"} Oct 31 13:34:51.682629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.680261 5532 scope.go:122] "RemoveContainer" containerID="b32b3ab39d686004880790adf10c37dda14d770f2dd40e659e970b36d566ae50" Oct 31 13:34:51.682629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.680415 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt" Oct 31 13:34:51.685691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.685569 5532 generic.go:356] "Generic (PLEG): container finished" podID="ededb5f2-22bb-4d1f-896a-63c8cc779422" containerID="d2b7d41ae02f7da641319599873650ee5a64ea18a9531ce5b3b6747c305b9405" exitCode=0 Oct 31 13:34:51.686115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.686076 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" event={"ID":"ededb5f2-22bb-4d1f-896a-63c8cc779422","Type":"ContainerDied","Data":"d2b7d41ae02f7da641319599873650ee5a64ea18a9531ce5b3b6747c305b9405"} Oct 31 13:34:51.686320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.686301 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" event={"ID":"ededb5f2-22bb-4d1f-896a-63c8cc779422","Type":"ContainerDied","Data":"5ad20d4548f6f3be7bc40ed24d89b74a46debaf887293c5f1d8c8853733c2d90"} Oct 31 13:34:51.686499 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.686486 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4" Oct 31 13:34:51.700637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.691474 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" event={"ID":"ceb475d1-1432-4095-b9c1-e6b16f18079a","Type":"ContainerStarted","Data":"25b19ee1348f14def41c88886f5f3f19776f83768175a4338b4b0819ce388dca"} Oct 31 13:34:51.700637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.695354 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" event={"ID":"83821ff1-96c3-41a0-a3e7-4940a2c1bb76","Type":"ContainerStarted","Data":"3dd8f197f0c013999e31377a1d078d2d0330ee1d92ae773e2e941bcad99daf30"} Oct 31 13:34:51.701486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.701444 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ededb5f2-22bb-4d1f-896a-63c8cc779422-kube-api-access-hh84x" pod "ededb5f2-22bb-4d1f-896a-63c8cc779422" (UID: "ededb5f2-22bb-4d1f-896a-63c8cc779422"). InnerVolumeSpecName "kube-api-access-hh84x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:34:51.707048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.706955 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:34:51.710109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.708723 5532 patch_prober.go:29] interesting pod/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 13:34:51.710109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.708775 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 13:34:51.763196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.755319 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrrvk\" (UniqueName: \"kubernetes.io/projected/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2-kube-api-access-hrrvk\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:34:51.763196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.755353 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh84x\" (UniqueName: \"kubernetes.io/projected/ededb5f2-22bb-4d1f-896a-63c8cc779422-kube-api-access-hh84x\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:34:51.789149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.788149 5532 scope.go:122] "RemoveContainer" containerID="b32b3ab39d686004880790adf10c37dda14d770f2dd40e659e970b36d566ae50" Oct 31 13:34:51.790053 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:34:51.789649 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b32b3ab39d686004880790adf10c37dda14d770f2dd40e659e970b36d566ae50\": container with ID starting with b32b3ab39d686004880790adf10c37dda14d770f2dd40e659e970b36d566ae50 not found: ID does not exist" containerID="b32b3ab39d686004880790adf10c37dda14d770f2dd40e659e970b36d566ae50" Oct 31 13:34:51.790053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.789692 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b32b3ab39d686004880790adf10c37dda14d770f2dd40e659e970b36d566ae50"} err="failed to get container status \"b32b3ab39d686004880790adf10c37dda14d770f2dd40e659e970b36d566ae50\": rpc error: code = NotFound desc = could not find container \"b32b3ab39d686004880790adf10c37dda14d770f2dd40e659e970b36d566ae50\": container with ID starting with b32b3ab39d686004880790adf10c37dda14d770f2dd40e659e970b36d566ae50 not found: ID does not exist" Oct 31 13:34:51.790053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.789722 5532 scope.go:122] "RemoveContainer" containerID="d2b7d41ae02f7da641319599873650ee5a64ea18a9531ce5b3b6747c305b9405" Oct 31 13:34:51.905395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.905360 5532 scope.go:122] "RemoveContainer" containerID="d2b7d41ae02f7da641319599873650ee5a64ea18a9531ce5b3b6747c305b9405" Oct 31 13:34:51.928616 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:34:51.906447 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2b7d41ae02f7da641319599873650ee5a64ea18a9531ce5b3b6747c305b9405\": container with ID starting with d2b7d41ae02f7da641319599873650ee5a64ea18a9531ce5b3b6747c305b9405 not found: ID does not exist" containerID="d2b7d41ae02f7da641319599873650ee5a64ea18a9531ce5b3b6747c305b9405" Oct 31 13:34:51.928954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:51.928908 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2b7d41ae02f7da641319599873650ee5a64ea18a9531ce5b3b6747c305b9405"} err="failed to get container status \"d2b7d41ae02f7da641319599873650ee5a64ea18a9531ce5b3b6747c305b9405\": rpc error: code = NotFound desc = could not find container \"d2b7d41ae02f7da641319599873650ee5a64ea18a9531ce5b3b6747c305b9405\": container with ID starting with d2b7d41ae02f7da641319599873650ee5a64ea18a9531ce5b3b6747c305b9405 not found: ID does not exist" Oct 31 13:34:52.005667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.003194 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt"] Oct 31 13:34:52.007991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.007954 5532 patch_prober.go:29] interesting pod/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:34:52.008141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.008022 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 13:34:52.020625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.020547 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:34:52.020625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.020609 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-fb8dt"] Oct 31 13:34:52.030220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.030188 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9"] Oct 31 13:34:52.034991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.034966 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4"] Oct 31 13:34:52.046674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.046645 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0e79e99-thmt9"] Oct 31 13:34:52.094793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.094759 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-79f4fb0f-5fc65ff467-mmgr4"] Oct 31 13:34:52.614630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.607949 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd"] Oct 31 13:34:52.614630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.609416 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ededb5f2-22bb-4d1f-896a-63c8cc779422" containerName="payload-pre-processing" Oct 31 13:34:52.614630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.609440 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ededb5f2-22bb-4d1f-896a-63c8cc779422" containerName="payload-pre-processing" Oct 31 13:34:52.614630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.609492 5532 cpu_manager.go:414] "containerMap: removing container" podUID="7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2" containerName="payload-processing" Oct 31 13:34:52.614630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.609501 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2" containerName="payload-processing" Oct 31 13:34:52.632709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.630616 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ededb5f2-22bb-4d1f-896a-63c8cc779422" containerName="payload-pre-processing" Oct 31 13:34:52.632709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.630682 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2" containerName="payload-processing" Oct 31 13:34:52.753795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.703959 5532 scope.go:122] "RemoveContainer" containerID="1a886f765485a6be1075519aeba3ff4b3e8164ac95fff22ff3981b8e168e9a55" Oct 31 13:34:52.935768 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.918272 5532 patch_prober.go:29] interesting pod/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 13:34:52.935768 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.918379 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 13:34:52.995754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.995714 5532 patch_prober.go:29] interesting pod/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:34:52.996046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:52.996015 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 13:34:53.007411 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.007381 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd"] Oct 31 13:34:53.007659 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.007487 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.016623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.016567 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cce3e7-serving-cert\"" Oct 31 13:34:53.136892 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.136855 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2aaa6e-c071-4d13-8aa9-857383962fdb-openshift-service-ca\") pod \"maas-api-e2e-api-b-cce3e7-764848994d-r7nxd\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.137053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.136916 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj6l7\" (UniqueName: \"kubernetes.io/projected/6e2aaa6e-c071-4d13-8aa9-857383962fdb-kube-api-access-lj6l7\") pod \"maas-api-e2e-api-b-cce3e7-764848994d-r7nxd\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.137222 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.137196 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-metrics-certs\") pod \"maas-api-e2e-api-b-cce3e7-764848994d-r7nxd\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.137806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.137780 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-maas-api-tls\") pod \"maas-api-e2e-api-b-cce3e7-764848994d-r7nxd\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.246674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.239806 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-maas-api-tls\") pod \"maas-api-e2e-api-b-cce3e7-764848994d-r7nxd\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.246674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.239926 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2aaa6e-c071-4d13-8aa9-857383962fdb-openshift-service-ca\") pod \"maas-api-e2e-api-b-cce3e7-764848994d-r7nxd\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.246674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.239990 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj6l7\" (UniqueName: \"kubernetes.io/projected/6e2aaa6e-c071-4d13-8aa9-857383962fdb-kube-api-access-lj6l7\") pod \"maas-api-e2e-api-b-cce3e7-764848994d-r7nxd\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.266552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.266513 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2aaa6e-c071-4d13-8aa9-857383962fdb-openshift-service-ca\") pod \"maas-api-e2e-api-b-cce3e7-764848994d-r7nxd\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.279468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.278986 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-metrics-certs\") pod \"maas-api-e2e-api-b-cce3e7-764848994d-r7nxd\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.304616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.303701 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-maas-api-tls\") pod \"maas-api-e2e-api-b-cce3e7-764848994d-r7nxd\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.304616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.304296 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-metrics-certs\") pod \"maas-api-e2e-api-b-cce3e7-764848994d-r7nxd\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.328342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.328310 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj6l7\" (UniqueName: \"kubernetes.io/projected/6e2aaa6e-c071-4d13-8aa9-857383962fdb-kube-api-access-lj6l7\") pod \"maas-api-e2e-api-b-cce3e7-764848994d-r7nxd\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.639666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.629348 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:53.799620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.791775 5532 patch_prober.go:29] interesting pod/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 13:34:53.799620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.791848 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 13:34:53.998061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.998020 5532 patch_prober.go:29] interesting pod/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:34:53.998222 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:53.998086 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 13:34:54.013901 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:54.013847 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2" path="/var/lib/kubelet/pods/7f5d90d8-494d-4be6-81a8-d8fc27e5f8d2/volumes" Oct 31 13:34:54.014898 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:54.014867 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5afab8a-9529-4205-98b3-8ea642c7ddeb" path="/var/lib/kubelet/pods/e5afab8a-9529-4205-98b3-8ea642c7ddeb/volumes" Oct 31 13:34:54.015676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:54.015652 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ededb5f2-22bb-4d1f-896a-63c8cc779422" path="/var/lib/kubelet/pods/ededb5f2-22bb-4d1f-896a-63c8cc779422/volumes" Oct 31 13:34:54.101745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:54.099586 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd"] Oct 31 13:34:54.104609 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:34:54.104564 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2aaa6e_c071_4d13_8aa9_857383962fdb.slice/crio-cdc93e2a6633bbdb6e40e8121aa35a0401aa889d0673b1a80ec6d50b12308b48 WatchSource:0}: Error finding container cdc93e2a6633bbdb6e40e8121aa35a0401aa889d0673b1a80ec6d50b12308b48: Status 404 returned error can't find the container with id cdc93e2a6633bbdb6e40e8121aa35a0401aa889d0673b1a80ec6d50b12308b48 Oct 31 13:34:54.790623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:54.717914 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" Oct 31 13:34:54.790623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:54.776218 5532 patch_prober.go:29] interesting pod/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 13:34:54.790623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:54.776448 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 13:34:54.998625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:54.938699 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4"] Oct 31 13:34:55.052914 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.044907 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4"] Oct 31 13:34:55.052914 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.046003 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" Oct 31 13:34:55.054162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.054115 5532 patch_prober.go:29] interesting pod/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:34:55.054410 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.054374 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 13:34:55.063296 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.063275 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-580f24-dockercfg-pb5zd\"" Oct 31 13:34:55.070222 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.070157 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-580f24\"" Oct 31 13:34:55.124373 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.124334 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" event={"ID":"6e2aaa6e-c071-4d13-8aa9-857383962fdb","Type":"ContainerStarted","Data":"cdc93e2a6633bbdb6e40e8121aa35a0401aa889d0673b1a80ec6d50b12308b48"} Oct 31 13:34:55.161660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.147672 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prwz\" (UniqueName: \"kubernetes.io/projected/7e9adade-cc78-4181-b53e-231fd46f9c7f-kube-api-access-7prwz\") pod \"payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4\" (UID: \"7e9adade-cc78-4181-b53e-231fd46f9c7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" Oct 31 13:34:55.161660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.148060 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9adade-cc78-4181-b53e-231fd46f9c7f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4\" (UID: \"7e9adade-cc78-4181-b53e-231fd46f9c7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" Oct 31 13:34:55.259953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.257923 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9adade-cc78-4181-b53e-231fd46f9c7f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4\" (UID: \"7e9adade-cc78-4181-b53e-231fd46f9c7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" Oct 31 13:34:55.259953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.258080 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prwz\" (UniqueName: \"kubernetes.io/projected/7e9adade-cc78-4181-b53e-231fd46f9c7f-kube-api-access-7prwz\") pod \"payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4\" (UID: \"7e9adade-cc78-4181-b53e-231fd46f9c7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" Oct 31 13:34:55.259953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.259425 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9adade-cc78-4181-b53e-231fd46f9c7f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4\" (UID: \"7e9adade-cc78-4181-b53e-231fd46f9c7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" Oct 31 13:34:55.292619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.283317 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prwz\" (UniqueName: \"kubernetes.io/projected/7e9adade-cc78-4181-b53e-231fd46f9c7f-kube-api-access-7prwz\") pod \"payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4\" (UID: \"7e9adade-cc78-4181-b53e-231fd46f9c7f\") " pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" Oct 31 13:34:55.418618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.417727 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" Oct 31 13:34:55.707146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.706516 5532 patch_prober.go:29] interesting pod/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 13:34:55.707146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.706589 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 13:34:55.833402 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:34:55.833368 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9adade_cc78_4181_b53e_231fd46f9c7f.slice/crio-3a7fd38c4efd797a54469eb54739bce14ea5902ba60db14f6daff7fbd967eb07 WatchSource:0}: Error finding container 3a7fd38c4efd797a54469eb54739bce14ea5902ba60db14f6daff7fbd967eb07: Status 404 returned error can't find the container with id 3a7fd38c4efd797a54469eb54739bce14ea5902ba60db14f6daff7fbd967eb07 Oct 31 13:34:55.834018 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.833995 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4"] Oct 31 13:34:55.996053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.995963 5532 patch_prober.go:29] interesting pod/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:34:55.996053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:55.996031 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 13:34:56.136282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:56.135956 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" event={"ID":"7e9adade-cc78-4181-b53e-231fd46f9c7f","Type":"ContainerStarted","Data":"e7bcddb2c2696b3b3e52618420d9633acce1403dd18d11d6e1acb2f07feef928"} Oct 31 13:34:56.136282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:56.136009 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" event={"ID":"7e9adade-cc78-4181-b53e-231fd46f9c7f","Type":"ContainerStarted","Data":"3a7fd38c4efd797a54469eb54739bce14ea5902ba60db14f6daff7fbd967eb07"} Oct 31 13:34:56.136282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:56.136044 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" Oct 31 13:34:56.138086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:56.138026 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" event={"ID":"6e2aaa6e-c071-4d13-8aa9-857383962fdb","Type":"ContainerStarted","Data":"5120043280e5153d95b77b388c43f479868ec905c1c14ac1b80e849ecc5d55a5"} Oct 31 13:34:56.138220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:56.138124 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:34:56.195619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:56.193248 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" podStartSLOduration=2.193227618 podStartE2EDuration="2.193227618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:56.170993353 +0000 UTC m=+1864.316737183" watchObservedRunningTime="2026-10-31 13:34:56.193227618 +0000 UTC m=+1864.338971449" Oct 31 13:34:56.222638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:56.221818 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" podStartSLOduration=3.230031275 podStartE2EDuration="4.221796909s" totalImagesPullingTime="991.765634ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:34:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:34:56.208002677 +0000 UTC m=+1864.353746505" watchObservedRunningTime="2026-10-31 13:34:56.221796909 +0000 UTC m=+1864.367540738" Oct 31 13:34:56.704656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:56.704621 5532 patch_prober.go:29] interesting pod/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Oct 31 13:34:56.704830 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:56.704674 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Oct 31 13:34:56.996115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:56.996032 5532 patch_prober.go:29] interesting pod/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Oct 31 13:34:56.996115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:56.996082 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Oct 31 13:34:57.388862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:57.388766 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:34:57.738364 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:57.738330 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:34:58.022466 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:58.022365 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:34:58.022902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:58.022706 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:34:58.023503 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:58.023475 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:34:58.176755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:58.176727 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:34:59.672969 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:59.672894 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z"] Oct 31 13:34:59.673874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:34:59.673161 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" podUID="c4ebfc88-1f62-43e9-bbf9-96216efffbe7" containerName="maas-api" containerID="cri-o://bb9502d420adf0cb85a93db5b3f9f683ef4a18548d940fbc5c0b436f020540fd" gracePeriod=30 Oct 31 13:35:00.134712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.133562 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm"] Oct 31 13:35:00.134712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.133636 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b06227e-z5gqm"] Oct 31 13:35:00.174031 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.174002 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:35:00.203293 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.203256 5532 generic.go:356] "Generic (PLEG): container finished" podID="c4ebfc88-1f62-43e9-bbf9-96216efffbe7" containerID="bb9502d420adf0cb85a93db5b3f9f683ef4a18548d940fbc5c0b436f020540fd" exitCode=0 Oct 31 13:35:00.203448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.203313 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" Oct 31 13:35:00.203448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.203336 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" event={"ID":"c4ebfc88-1f62-43e9-bbf9-96216efffbe7","Type":"ContainerDied","Data":"bb9502d420adf0cb85a93db5b3f9f683ef4a18548d940fbc5c0b436f020540fd"} Oct 31 13:35:00.203448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.203394 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z" event={"ID":"c4ebfc88-1f62-43e9-bbf9-96216efffbe7","Type":"ContainerDied","Data":"ae259a13216c97927d2caf7cd1ab37b6701e843a098b44d42069534236cae187"} Oct 31 13:35:00.203448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.203418 5532 scope.go:122] "RemoveContainer" containerID="bb9502d420adf0cb85a93db5b3f9f683ef4a18548d940fbc5c0b436f020540fd" Oct 31 13:35:00.217363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.217341 5532 scope.go:122] "RemoveContainer" containerID="bb9502d420adf0cb85a93db5b3f9f683ef4a18548d940fbc5c0b436f020540fd" Oct 31 13:35:00.217922 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:35:00.217875 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb9502d420adf0cb85a93db5b3f9f683ef4a18548d940fbc5c0b436f020540fd\": container with ID starting with bb9502d420adf0cb85a93db5b3f9f683ef4a18548d940fbc5c0b436f020540fd not found: ID does not exist" containerID="bb9502d420adf0cb85a93db5b3f9f683ef4a18548d940fbc5c0b436f020540fd" Oct 31 13:35:00.218085 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.217929 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb9502d420adf0cb85a93db5b3f9f683ef4a18548d940fbc5c0b436f020540fd"} err="failed to get container status \"bb9502d420adf0cb85a93db5b3f9f683ef4a18548d940fbc5c0b436f020540fd\": rpc error: code = NotFound desc = could not find container \"bb9502d420adf0cb85a93db5b3f9f683ef4a18548d940fbc5c0b436f020540fd\": container with ID starting with bb9502d420adf0cb85a93db5b3f9f683ef4a18548d940fbc5c0b436f020540fd not found: ID does not exist" Oct 31 13:35:00.247395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.247366 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-metrics-certs\") pod \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " Oct 31 13:35:00.247546 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.247406 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-maas-api-tls\") pod \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " Oct 31 13:35:00.247701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.247686 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-kube-api-access-dnqm7\" (UniqueName: \"kubernetes.io/projected/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-kube-api-access-dnqm7\") pod \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " Oct 31 13:35:00.247769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.247718 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-openshift-service-ca\") pod \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\" (UID: \"c4ebfc88-1f62-43e9-bbf9-96216efffbe7\") " Oct 31 13:35:00.248184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.248152 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-openshift-service-ca" pod "c4ebfc88-1f62-43e9-bbf9-96216efffbe7" (UID: "c4ebfc88-1f62-43e9-bbf9-96216efffbe7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:35:00.248647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.248578 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:00.249675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.249645 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-maas-api-tls" pod "c4ebfc88-1f62-43e9-bbf9-96216efffbe7" (UID: "c4ebfc88-1f62-43e9-bbf9-96216efffbe7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:35:00.249833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.249815 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-kube-api-access-dnqm7" pod "c4ebfc88-1f62-43e9-bbf9-96216efffbe7" (UID: "c4ebfc88-1f62-43e9-bbf9-96216efffbe7"). InnerVolumeSpecName "kube-api-access-dnqm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:00.250257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.250237 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-metrics-certs" pod "c4ebfc88-1f62-43e9-bbf9-96216efffbe7" (UID: "c4ebfc88-1f62-43e9-bbf9-96216efffbe7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:35:00.350219 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.350174 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:00.350219 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.350214 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:00.350219 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.350226 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnqm7\" (UniqueName: \"kubernetes.io/projected/c4ebfc88-1f62-43e9-bbf9-96216efffbe7-kube-api-access-dnqm7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:00.548326 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.548292 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z"] Oct 31 13:35:00.555149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.555104 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-d6k6z"] Oct 31 13:35:00.640400 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:00.640364 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz"] Oct 31 13:35:01.769069 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:35:01.769030 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:35:02.020269 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:02.020186 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="301589df-3ff3-4a56-a4cf-7ca0e1471852" path="/var/lib/kubelet/pods/301589df-3ff3-4a56-a4cf-7ca0e1471852/volumes" Oct 31 13:35:02.020791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:02.020775 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4ebfc88-1f62-43e9-bbf9-96216efffbe7" path="/var/lib/kubelet/pods/c4ebfc88-1f62-43e9-bbf9-96216efffbe7/volumes" Oct 31 13:35:02.173770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:02.173737 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:35:05.640934 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:05.640890 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" podUID="21375def-9f25-4f90-a449-2bdff0362119" containerName="payload-processing" containerID="cri-o://563ae7f49f8ff8a499259a3e9fdecb5cb8d3fc4f7dfecf9297cc7f243b3bdd46" gracePeriod=25 Oct 31 13:35:06.015784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.015756 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" Oct 31 13:35:06.175714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.175677 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21375def-9f25-4f90-a449-2bdff0362119-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21375def-9f25-4f90-a449-2bdff0362119-plugins-config-volume\") pod \"21375def-9f25-4f90-a449-2bdff0362119\" (UID: \"21375def-9f25-4f90-a449-2bdff0362119\") " Oct 31 13:35:06.175922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.175904 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21375def-9f25-4f90-a449-2bdff0362119-kube-api-access-qjlx2\" (UniqueName: \"kubernetes.io/projected/21375def-9f25-4f90-a449-2bdff0362119-kube-api-access-qjlx2\") pod \"21375def-9f25-4f90-a449-2bdff0362119\" (UID: \"21375def-9f25-4f90-a449-2bdff0362119\") " Oct 31 13:35:06.176178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.176149 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21375def-9f25-4f90-a449-2bdff0362119-plugins-config-volume" pod "21375def-9f25-4f90-a449-2bdff0362119" (UID: "21375def-9f25-4f90-a449-2bdff0362119"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:35:06.176864 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.176832 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21375def-9f25-4f90-a449-2bdff0362119-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:06.178053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.178031 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21375def-9f25-4f90-a449-2bdff0362119-kube-api-access-qjlx2" pod "21375def-9f25-4f90-a449-2bdff0362119" (UID: "21375def-9f25-4f90-a449-2bdff0362119"). InnerVolumeSpecName "kube-api-access-qjlx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:06.278882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.278796 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjlx2\" (UniqueName: \"kubernetes.io/projected/21375def-9f25-4f90-a449-2bdff0362119-kube-api-access-qjlx2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:06.292888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.292855 5532 generic.go:356] "Generic (PLEG): container finished" podID="21375def-9f25-4f90-a449-2bdff0362119" containerID="563ae7f49f8ff8a499259a3e9fdecb5cb8d3fc4f7dfecf9297cc7f243b3bdd46" exitCode=0 Oct 31 13:35:06.293060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.292918 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" Oct 31 13:35:06.293060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.292927 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" event={"ID":"21375def-9f25-4f90-a449-2bdff0362119","Type":"ContainerDied","Data":"563ae7f49f8ff8a499259a3e9fdecb5cb8d3fc4f7dfecf9297cc7f243b3bdd46"} Oct 31 13:35:06.293060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.292982 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz" event={"ID":"21375def-9f25-4f90-a449-2bdff0362119","Type":"ContainerDied","Data":"22fe651ea4dfd08fcffdeddd06a03ed9397521668d826fa21e76059535d1e05d"} Oct 31 13:35:06.293060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.293013 5532 scope.go:122] "RemoveContainer" containerID="563ae7f49f8ff8a499259a3e9fdecb5cb8d3fc4f7dfecf9297cc7f243b3bdd46" Oct 31 13:35:06.304870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.304842 5532 scope.go:122] "RemoveContainer" containerID="563ae7f49f8ff8a499259a3e9fdecb5cb8d3fc4f7dfecf9297cc7f243b3bdd46" Oct 31 13:35:06.305140 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:35:06.305121 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563ae7f49f8ff8a499259a3e9fdecb5cb8d3fc4f7dfecf9297cc7f243b3bdd46\": container with ID starting with 563ae7f49f8ff8a499259a3e9fdecb5cb8d3fc4f7dfecf9297cc7f243b3bdd46 not found: ID does not exist" containerID="563ae7f49f8ff8a499259a3e9fdecb5cb8d3fc4f7dfecf9297cc7f243b3bdd46" Oct 31 13:35:06.305203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.305154 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563ae7f49f8ff8a499259a3e9fdecb5cb8d3fc4f7dfecf9297cc7f243b3bdd46"} err="failed to get container status \"563ae7f49f8ff8a499259a3e9fdecb5cb8d3fc4f7dfecf9297cc7f243b3bdd46\": rpc error: code = NotFound desc = could not find container \"563ae7f49f8ff8a499259a3e9fdecb5cb8d3fc4f7dfecf9297cc7f243b3bdd46\": container with ID starting with 563ae7f49f8ff8a499259a3e9fdecb5cb8d3fc4f7dfecf9297cc7f243b3bdd46 not found: ID does not exist" Oct 31 13:35:06.323718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.323688 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz"] Oct 31 13:35:06.331260 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:06.331234 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-466002-7d68dc4d5f-h7pwz"] Oct 31 13:35:07.170427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.170399 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" Oct 31 13:35:07.491956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.491877 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp"] Oct 31 13:35:07.492725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.492700 5532 cpu_manager.go:414] "containerMap: removing container" podUID="21375def-9f25-4f90-a449-2bdff0362119" containerName="payload-processing" Oct 31 13:35:07.492725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.492719 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21375def-9f25-4f90-a449-2bdff0362119" containerName="payload-processing" Oct 31 13:35:07.492929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.492782 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c4ebfc88-1f62-43e9-bbf9-96216efffbe7" containerName="maas-api" Oct 31 13:35:07.492929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.492790 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4ebfc88-1f62-43e9-bbf9-96216efffbe7" containerName="maas-api" Oct 31 13:35:07.493032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.493016 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4ebfc88-1f62-43e9-bbf9-96216efffbe7" containerName="maas-api" Oct 31 13:35:07.493086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.493045 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="21375def-9f25-4f90-a449-2bdff0362119" containerName="payload-processing" Oct 31 13:35:07.499832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.499262 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:07.521657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.516342 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp"] Oct 31 13:35:07.521657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.517506 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-580f24-serving-cert\"" Oct 31 13:35:07.602767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.602728 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9040d659-6f09-4fc7-a4b3-193870beb6c2-openshift-service-ca\") pod \"maas-api-e2e-api-a-580f24-dc944ddfb-fjssp\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:07.602963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.602774 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-metrics-certs\") pod \"maas-api-e2e-api-a-580f24-dc944ddfb-fjssp\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:07.602963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.602925 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78xlc\" (UniqueName: \"kubernetes.io/projected/9040d659-6f09-4fc7-a4b3-193870beb6c2-kube-api-access-78xlc\") pod \"maas-api-e2e-api-a-580f24-dc944ddfb-fjssp\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:07.603079 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.603011 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-maas-api-tls\") pod \"maas-api-e2e-api-a-580f24-dc944ddfb-fjssp\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:07.704976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.704938 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9040d659-6f09-4fc7-a4b3-193870beb6c2-openshift-service-ca\") pod \"maas-api-e2e-api-a-580f24-dc944ddfb-fjssp\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:07.705174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.704979 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-metrics-certs\") pod \"maas-api-e2e-api-a-580f24-dc944ddfb-fjssp\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:07.705174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.705136 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78xlc\" (UniqueName: \"kubernetes.io/projected/9040d659-6f09-4fc7-a4b3-193870beb6c2-kube-api-access-78xlc\") pod \"maas-api-e2e-api-a-580f24-dc944ddfb-fjssp\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:07.705292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.705222 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-maas-api-tls\") pod \"maas-api-e2e-api-a-580f24-dc944ddfb-fjssp\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:07.705822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.705789 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9040d659-6f09-4fc7-a4b3-193870beb6c2-openshift-service-ca\") pod \"maas-api-e2e-api-a-580f24-dc944ddfb-fjssp\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:07.707910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.707880 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-metrics-certs\") pod \"maas-api-e2e-api-a-580f24-dc944ddfb-fjssp\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:07.708036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.707951 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-maas-api-tls\") pod \"maas-api-e2e-api-a-580f24-dc944ddfb-fjssp\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:07.718421 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.718392 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78xlc\" (UniqueName: \"kubernetes.io/projected/9040d659-6f09-4fc7-a4b3-193870beb6c2-kube-api-access-78xlc\") pod \"maas-api-e2e-api-a-580f24-dc944ddfb-fjssp\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:07.842604 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:07.842506 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:08.013196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.013163 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21375def-9f25-4f90-a449-2bdff0362119" path="/var/lib/kubelet/pods/21375def-9f25-4f90-a449-2bdff0362119/volumes" Oct 31 13:35:08.135384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.135356 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp"] Oct 31 13:35:08.137743 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:35:08.137712 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9040d659_6f09_4fc7_a4b3_193870beb6c2.slice/crio-ed32aff5806a63adff5d2f219c4141311180f9046743d905d09da52a91097530 WatchSource:0}: Error finding container ed32aff5806a63adff5d2f219c4141311180f9046743d905d09da52a91097530: Status 404 returned error can't find the container with id ed32aff5806a63adff5d2f219c4141311180f9046743d905d09da52a91097530 Oct 31 13:35:08.301348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.301309 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h"] Oct 31 13:35:08.307921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.307895 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" Oct 31 13:35:08.316393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.316368 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h"] Oct 31 13:35:08.348651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.348618 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-79f4fb0f-dockercfg-zc24t\"" Oct 31 13:35:08.348845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.348754 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-79f4fb0f\"" Oct 31 13:35:08.402654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.402563 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" event={"ID":"9040d659-6f09-4fc7-a4b3-193870beb6c2","Type":"ContainerStarted","Data":"ed32aff5806a63adff5d2f219c4141311180f9046743d905d09da52a91097530"} Oct 31 13:35:08.428647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.420832 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qqfr\" (UniqueName: \"kubernetes.io/projected/70bec635-576b-4a02-999e-c43a064633b0-kube-api-access-8qqfr\") pod \"payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h\" (UID: \"70bec635-576b-4a02-999e-c43a064633b0\") " pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" Oct 31 13:35:08.452981 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.428930 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bec635-576b-4a02-999e-c43a064633b0-plugins-config-volume\") pod \"payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h\" (UID: \"70bec635-576b-4a02-999e-c43a064633b0\") " pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" Oct 31 13:35:08.536620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.531181 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qqfr\" (UniqueName: \"kubernetes.io/projected/70bec635-576b-4a02-999e-c43a064633b0-kube-api-access-8qqfr\") pod \"payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h\" (UID: \"70bec635-576b-4a02-999e-c43a064633b0\") " pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" Oct 31 13:35:08.536620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.531371 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bec635-576b-4a02-999e-c43a064633b0-plugins-config-volume\") pod \"payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h\" (UID: \"70bec635-576b-4a02-999e-c43a064633b0\") " pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" Oct 31 13:35:08.536620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.532456 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bec635-576b-4a02-999e-c43a064633b0-plugins-config-volume\") pod \"payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h\" (UID: \"70bec635-576b-4a02-999e-c43a064633b0\") " pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" Oct 31 13:35:08.577614 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.574511 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qqfr\" (UniqueName: \"kubernetes.io/projected/70bec635-576b-4a02-999e-c43a064633b0-kube-api-access-8qqfr\") pod \"payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h\" (UID: \"70bec635-576b-4a02-999e-c43a064633b0\") " pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" Oct 31 13:35:08.639185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.628094 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" Oct 31 13:35:08.945698 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:08.944014 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h"] Oct 31 13:35:08.946958 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:35:08.946918 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bec635_576b_4a02_999e_c43a064633b0.slice/crio-09a9ec8833b2f439cc98fde3c0b733fe942a29194510b0e529419f8ebfb2e872 WatchSource:0}: Error finding container 09a9ec8833b2f439cc98fde3c0b733fe942a29194510b0e529419f8ebfb2e872: Status 404 returned error can't find the container with id 09a9ec8833b2f439cc98fde3c0b733fe942a29194510b0e529419f8ebfb2e872 Oct 31 13:35:09.413302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:09.413249 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" event={"ID":"70bec635-576b-4a02-999e-c43a064633b0","Type":"ContainerStarted","Data":"9219fffc1b70e1390dafed4aefd2e0210697f5c636b768c2df3d7dcc7fdaf490"} Oct 31 13:35:09.413776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:09.413316 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" event={"ID":"70bec635-576b-4a02-999e-c43a064633b0","Type":"ContainerStarted","Data":"09a9ec8833b2f439cc98fde3c0b733fe942a29194510b0e529419f8ebfb2e872"} Oct 31 13:35:09.413776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:09.413347 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" Oct 31 13:35:09.415257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:09.415232 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" event={"ID":"9040d659-6f09-4fc7-a4b3-193870beb6c2","Type":"ContainerStarted","Data":"364ac701927cba1693f5f452bfb15fd8685fce01c0dfaeb360fc6c0dd1807dd6"} Oct 31 13:35:09.415407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:09.415367 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:09.456235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:09.456189 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" podStartSLOduration=1.456174688 podStartE2EDuration="1.456174688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:09.45140662 +0000 UTC m=+1877.597150450" watchObservedRunningTime="2026-10-31 13:35:09.456174688 +0000 UTC m=+1877.601918518" Oct 31 13:35:09.476122 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:09.476068 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" podStartSLOduration=1.9728143569999999 podStartE2EDuration="2.476050866s" totalImagesPullingTime="503.236509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:09.475356057 +0000 UTC m=+1877.621099883" watchObservedRunningTime="2026-10-31 13:35:09.476050866 +0000 UTC m=+1877.621794695" Oct 31 13:35:13.031450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:13.031414 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz"] Oct 31 13:35:13.035677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:13.035652 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64c88169-gkhzz"] Oct 31 13:35:14.006187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:14.006154 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbca6d33-4441-4d6c-94cb-b9fa65689083" path="/var/lib/kubelet/pods/bbca6d33-4441-4d6c-94cb-b9fa65689083/volumes" Oct 31 13:35:15.430608 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.430562 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:35:15.537013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.536984 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67"] Oct 31 13:35:15.542892 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.542867 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:15.551163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.551134 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67"] Oct 31 13:35:15.551358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.551201 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-79f4fb0f-serving-cert\"" Oct 31 13:35:15.636430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.636385 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-metrics-certs\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:15.636674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.636652 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/692f58c7-b8b9-4ca5-aedc-8f4a74804549-openshift-service-ca\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:15.636744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.636678 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-maas-api-tls\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:15.636977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.636947 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g7zr\" (UniqueName: \"kubernetes.io/projected/692f58c7-b8b9-4ca5-aedc-8f4a74804549-kube-api-access-6g7zr\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:15.739021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.738936 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-metrics-certs\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:15.739503 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.739475 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/692f58c7-b8b9-4ca5-aedc-8f4a74804549-openshift-service-ca\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:15.739643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.739626 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-maas-api-tls\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:15.739941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.739925 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g7zr\" (UniqueName: \"kubernetes.io/projected/692f58c7-b8b9-4ca5-aedc-8f4a74804549-kube-api-access-6g7zr\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:15.741229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.741204 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/692f58c7-b8b9-4ca5-aedc-8f4a74804549-openshift-service-ca\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:15.741415 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:35:15.741402 5532 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-serving-cert: secret "maas-api-e2e-mt-79f4fb0f-serving-cert" not found Oct 31 13:35:15.741541 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:35:15.741531 5532 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-maas-api-tls podName:692f58c7-b8b9-4ca5-aedc-8f4a74804549 nodeName:}" failed. No retries permitted until 2026-10-31 13:35:16.241512677 +0000 UTC m=+1884.387256496 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-maas-api-tls") pod "maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" (UID: "692f58c7-b8b9-4ca5-aedc-8f4a74804549") : secret "maas-api-e2e-mt-79f4fb0f-serving-cert" not found Oct 31 13:35:15.746235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.746177 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-metrics-certs\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:15.759757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:15.759667 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g7zr\" (UniqueName: \"kubernetes.io/projected/692f58c7-b8b9-4ca5-aedc-8f4a74804549-kube-api-access-6g7zr\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:16.250319 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:16.250281 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-maas-api-tls\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:16.253068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:16.253039 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-maas-api-tls\") pod \"maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:16.481703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:16.481673 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:16.853986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:16.853949 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67"] Oct 31 13:35:16.855913 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:35:16.855887 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod692f58c7_b8b9_4ca5_aedc_8f4a74804549.slice/crio-c8ce57bbd86d9bc9ad08d69676441b9467cdc555bed24ee1c2249326483434b5 WatchSource:0}: Error finding container c8ce57bbd86d9bc9ad08d69676441b9467cdc555bed24ee1c2249326483434b5: Status 404 returned error can't find the container with id c8ce57bbd86d9bc9ad08d69676441b9467cdc555bed24ee1c2249326483434b5 Oct 31 13:35:17.495850 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:17.495809 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" event={"ID":"692f58c7-b8b9-4ca5-aedc-8f4a74804549","Type":"ContainerStarted","Data":"0f2f8a327b0117b9e7c76d7ee02c6a0df143e0a4616f363c9571c3d869d4c260"} Oct 31 13:35:17.496248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:17.495874 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" event={"ID":"692f58c7-b8b9-4ca5-aedc-8f4a74804549","Type":"ContainerStarted","Data":"c8ce57bbd86d9bc9ad08d69676441b9467cdc555bed24ee1c2249326483434b5"} Oct 31 13:35:17.496248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:17.495914 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:17.521367 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:17.521260 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" podStartSLOduration=2.112512248 podStartE2EDuration="2.521241516s" totalImagesPullingTime="408.729268ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:17.518864093 +0000 UTC m=+1885.664607926" watchObservedRunningTime="2026-10-31 13:35:17.521241516 +0000 UTC m=+1885.666985346" Oct 31 13:35:18.033691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:18.033654 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d"] Oct 31 13:35:18.041865 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:18.041837 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74eeca55-qqz9d"] Oct 31 13:35:20.007541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:20.007504 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e" path="/var/lib/kubelet/pods/fa6d4f64-a203-4e4f-9a12-b9fdd4bfe31e/volumes" Oct 31 13:35:20.007946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:20.007924 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:35:20.428514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:20.428482 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" Oct 31 13:35:23.035554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:23.035489 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8"] Oct 31 13:35:23.066900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:23.066844 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f986406f-t6rj8"] Oct 31 13:35:23.515630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:23.514026 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:35:24.012989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:24.012947 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9799050e-794b-4634-9838-503737ce1cd0" path="/var/lib/kubelet/pods/9799050e-794b-4634-9838-503737ce1cd0/volumes" Oct 31 13:35:28.046551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:28.046517 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82"] Oct 31 13:35:28.073118 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:28.073084 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w"] Oct 31 13:35:29.031274 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:29.031223 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7"] Oct 31 13:35:29.039371 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:29.039333 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73999081-68vp7"] Oct 31 13:35:30.006927 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:30.006847 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f961cdee-55f0-4b63-b43e-0394108e51d1" path="/var/lib/kubelet/pods/f961cdee-55f0-4b63-b43e-0394108e51d1/volumes" Oct 31 13:35:31.031498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:31.031460 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278"] Oct 31 13:35:31.035753 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:31.035724 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ceeaf5ef-72278"] Oct 31 13:35:31.768168 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:35:31.768134 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:35:32.007006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:32.006964 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45325bfe-233f-47ca-8457-8f6b197ec554" path="/var/lib/kubelet/pods/45325bfe-233f-47ca-8457-8f6b197ec554/volumes" Oct 31 13:35:32.065429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:32.065341 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5"] Oct 31 13:35:32.069536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:32.069511 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk"] Oct 31 13:35:32.073120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:32.073101 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17c30975-bnqmk"] Oct 31 13:35:32.076539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:32.076520 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2680b5e-kjqs5"] Oct 31 13:35:33.046848 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.046805 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" podUID="76cc343f-d6c3-4df5-95b4-4384bd95e2d8" containerName="payload-processing" containerID="cri-o://aec8ddd7a42607a5eafe34137df9d20f929129d2f1860669fb2a77362933bd35" gracePeriod=25 Oct 31 13:35:33.083906 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.073809 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" podUID="f031a89b-1620-4cf6-a62c-4d73bce14623" containerName="payload-pre-processing" containerID="cri-o://211dfa39dcec5b0175f589f82e753f4087d87a6b953d43ededa5e7ab6fe73915" gracePeriod=25 Oct 31 13:35:33.437968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.437804 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" Oct 31 13:35:33.543784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.543756 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" Oct 31 13:35:33.604067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.604038 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-plugins-config-volume\") pod \"76cc343f-d6c3-4df5-95b4-4384bd95e2d8\" (UID: \"76cc343f-d6c3-4df5-95b4-4384bd95e2d8\") " Oct 31 13:35:33.604266 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.604168 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-kube-api-access-wb9gj\" (UniqueName: \"kubernetes.io/projected/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-kube-api-access-wb9gj\") pod \"76cc343f-d6c3-4df5-95b4-4384bd95e2d8\" (UID: \"76cc343f-d6c3-4df5-95b4-4384bd95e2d8\") " Oct 31 13:35:33.604534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.604507 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-plugins-config-volume" pod "76cc343f-d6c3-4df5-95b4-4384bd95e2d8" (UID: "76cc343f-d6c3-4df5-95b4-4384bd95e2d8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:35:33.605208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.605179 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:33.615177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.615149 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-kube-api-access-wb9gj" pod "76cc343f-d6c3-4df5-95b4-4384bd95e2d8" (UID: "76cc343f-d6c3-4df5-95b4-4384bd95e2d8"). InnerVolumeSpecName "kube-api-access-wb9gj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:33.706693 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.706663 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f031a89b-1620-4cf6-a62c-4d73bce14623-kube-api-access-pktcw\" (UniqueName: \"kubernetes.io/projected/f031a89b-1620-4cf6-a62c-4d73bce14623-kube-api-access-pktcw\") pod \"f031a89b-1620-4cf6-a62c-4d73bce14623\" (UID: \"f031a89b-1620-4cf6-a62c-4d73bce14623\") " Oct 31 13:35:33.706881 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.706866 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f031a89b-1620-4cf6-a62c-4d73bce14623-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f031a89b-1620-4cf6-a62c-4d73bce14623-plugins-config-volume\") pod \"f031a89b-1620-4cf6-a62c-4d73bce14623\" (UID: \"f031a89b-1620-4cf6-a62c-4d73bce14623\") " Oct 31 13:35:33.707281 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.707256 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f031a89b-1620-4cf6-a62c-4d73bce14623-plugins-config-volume" pod "f031a89b-1620-4cf6-a62c-4d73bce14623" (UID: "f031a89b-1620-4cf6-a62c-4d73bce14623"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:35:33.707806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.707785 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb9gj\" (UniqueName: \"kubernetes.io/projected/76cc343f-d6c3-4df5-95b4-4384bd95e2d8-kube-api-access-wb9gj\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:33.707891 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.707807 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f031a89b-1620-4cf6-a62c-4d73bce14623-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:33.709044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.709025 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f031a89b-1620-4cf6-a62c-4d73bce14623-kube-api-access-pktcw" pod "f031a89b-1620-4cf6-a62c-4d73bce14623" (UID: "f031a89b-1620-4cf6-a62c-4d73bce14623"). InnerVolumeSpecName "kube-api-access-pktcw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:33.809296 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.809258 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pktcw\" (UniqueName: \"kubernetes.io/projected/f031a89b-1620-4cf6-a62c-4d73bce14623-kube-api-access-pktcw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:33.809296 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.809280 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" event={"ID":"76cc343f-d6c3-4df5-95b4-4384bd95e2d8","Type":"ContainerDied","Data":"aec8ddd7a42607a5eafe34137df9d20f929129d2f1860669fb2a77362933bd35"} Oct 31 13:35:33.809514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.809314 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" Oct 31 13:35:33.809514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.809351 5532 scope.go:122] "RemoveContainer" containerID="aec8ddd7a42607a5eafe34137df9d20f929129d2f1860669fb2a77362933bd35" Oct 31 13:35:33.809514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.809260 5532 generic.go:356] "Generic (PLEG): container finished" podID="76cc343f-d6c3-4df5-95b4-4384bd95e2d8" containerID="aec8ddd7a42607a5eafe34137df9d20f929129d2f1860669fb2a77362933bd35" exitCode=0 Oct 31 13:35:33.809514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.809455 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82" event={"ID":"76cc343f-d6c3-4df5-95b4-4384bd95e2d8","Type":"ContainerDied","Data":"3b24534c01822925791c0bf67620df2c158ecdc1c62a46adee344222635d7e36"} Oct 31 13:35:33.811031 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.811008 5532 generic.go:356] "Generic (PLEG): container finished" podID="f031a89b-1620-4cf6-a62c-4d73bce14623" containerID="211dfa39dcec5b0175f589f82e753f4087d87a6b953d43ededa5e7ab6fe73915" exitCode=0 Oct 31 13:35:33.811171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.811037 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" event={"ID":"f031a89b-1620-4cf6-a62c-4d73bce14623","Type":"ContainerDied","Data":"211dfa39dcec5b0175f589f82e753f4087d87a6b953d43ededa5e7ab6fe73915"} Oct 31 13:35:33.811171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.811066 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" Oct 31 13:35:33.811171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.811083 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w" event={"ID":"f031a89b-1620-4cf6-a62c-4d73bce14623","Type":"ContainerDied","Data":"4a77625a66fdfe528e28bbaffe9b9b958a9c1f60fc7a56772cca2f802686cc45"} Oct 31 13:35:33.822776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.822759 5532 scope.go:122] "RemoveContainer" containerID="aec8ddd7a42607a5eafe34137df9d20f929129d2f1860669fb2a77362933bd35" Oct 31 13:35:33.823078 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:35:33.823056 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec8ddd7a42607a5eafe34137df9d20f929129d2f1860669fb2a77362933bd35\": container with ID starting with aec8ddd7a42607a5eafe34137df9d20f929129d2f1860669fb2a77362933bd35 not found: ID does not exist" containerID="aec8ddd7a42607a5eafe34137df9d20f929129d2f1860669fb2a77362933bd35" Oct 31 13:35:33.823178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.823086 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec8ddd7a42607a5eafe34137df9d20f929129d2f1860669fb2a77362933bd35"} err="failed to get container status \"aec8ddd7a42607a5eafe34137df9d20f929129d2f1860669fb2a77362933bd35\": rpc error: code = NotFound desc = could not find container \"aec8ddd7a42607a5eafe34137df9d20f929129d2f1860669fb2a77362933bd35\": container with ID starting with aec8ddd7a42607a5eafe34137df9d20f929129d2f1860669fb2a77362933bd35 not found: ID does not exist" Oct 31 13:35:33.823178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.823102 5532 scope.go:122] "RemoveContainer" containerID="211dfa39dcec5b0175f589f82e753f4087d87a6b953d43ededa5e7ab6fe73915" Oct 31 13:35:33.834527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.834513 5532 scope.go:122] "RemoveContainer" containerID="211dfa39dcec5b0175f589f82e753f4087d87a6b953d43ededa5e7ab6fe73915" Oct 31 13:35:33.834790 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:35:33.834769 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211dfa39dcec5b0175f589f82e753f4087d87a6b953d43ededa5e7ab6fe73915\": container with ID starting with 211dfa39dcec5b0175f589f82e753f4087d87a6b953d43ededa5e7ab6fe73915 not found: ID does not exist" containerID="211dfa39dcec5b0175f589f82e753f4087d87a6b953d43ededa5e7ab6fe73915" Oct 31 13:35:33.834863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.834795 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211dfa39dcec5b0175f589f82e753f4087d87a6b953d43ededa5e7ab6fe73915"} err="failed to get container status \"211dfa39dcec5b0175f589f82e753f4087d87a6b953d43ededa5e7ab6fe73915\": rpc error: code = NotFound desc = could not find container \"211dfa39dcec5b0175f589f82e753f4087d87a6b953d43ededa5e7ab6fe73915\": container with ID starting with 211dfa39dcec5b0175f589f82e753f4087d87a6b953d43ededa5e7ab6fe73915 not found: ID does not exist" Oct 31 13:35:33.841898 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.841878 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w"] Oct 31 13:35:33.846244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.846221 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-75c956-5ccd94c96c-ng27w"] Oct 31 13:35:33.858121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.858099 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82"] Oct 31 13:35:33.862369 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:33.862349 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-75c956-7d588d79fc-9kt82"] Oct 31 13:35:34.005811 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.005736 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76cc343f-d6c3-4df5-95b4-4384bd95e2d8" path="/var/lib/kubelet/pods/76cc343f-d6c3-4df5-95b4-4384bd95e2d8/volumes" Oct 31 13:35:34.006286 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.006271 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="874d4baf-d227-49f6-9b75-7e18e1a729f1" path="/var/lib/kubelet/pods/874d4baf-d227-49f6-9b75-7e18e1a729f1/volumes" Oct 31 13:35:34.006765 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.006749 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d58c3e46-103b-408d-a0a0-b4fd24891dfc" path="/var/lib/kubelet/pods/d58c3e46-103b-408d-a0a0-b4fd24891dfc/volumes" Oct 31 13:35:34.007479 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.007464 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f031a89b-1620-4cf6-a62c-4d73bce14623" path="/var/lib/kubelet/pods/f031a89b-1620-4cf6-a62c-4d73bce14623/volumes" Oct 31 13:35:34.450825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.450791 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns"] Oct 31 13:35:34.451870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.451844 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f031a89b-1620-4cf6-a62c-4d73bce14623" containerName="payload-pre-processing" Oct 31 13:35:34.451870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.451866 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f031a89b-1620-4cf6-a62c-4d73bce14623" containerName="payload-pre-processing" Oct 31 13:35:34.452055 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.451889 5532 cpu_manager.go:414] "containerMap: removing container" podUID="76cc343f-d6c3-4df5-95b4-4384bd95e2d8" containerName="payload-processing" Oct 31 13:35:34.452055 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.451898 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76cc343f-d6c3-4df5-95b4-4384bd95e2d8" containerName="payload-processing" Oct 31 13:35:34.452248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.452231 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f031a89b-1620-4cf6-a62c-4d73bce14623" containerName="payload-pre-processing" Oct 31 13:35:34.452315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.452254 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="76cc343f-d6c3-4df5-95b4-4384bd95e2d8" containerName="payload-processing" Oct 31 13:35:34.457942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.457917 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.471335 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.471311 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns"] Oct 31 13:35:34.528138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.528094 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-cbb3d5-openshift-default-dockercfg-6x7zg\"" Oct 31 13:35:34.776718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.735072 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-data\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.776718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.735128 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-podinfo\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.776718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.735174 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-envoy\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.776718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.735309 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-socket\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.776718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.735338 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-token\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.776718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.735406 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dnbm\" (UniqueName: \"kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-kube-api-access-5dnbm\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.776718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.735493 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-credential-socket\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.776718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.735516 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-certs\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.776718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.761642 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istiod-ca-cert\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.866831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.866794 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-data\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.867087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.866846 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-podinfo\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.867087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.866888 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-envoy\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.867087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.867014 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-socket\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.867087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.867039 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-token\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.867324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.867101 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dnbm\" (UniqueName: \"kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-kube-api-access-5dnbm\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.867324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.867184 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-credential-socket\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.867324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.867211 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-certs\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.867687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.867425 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istiod-ca-cert\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.869213 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.869185 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istiod-ca-cert\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.869491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.869471 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-data\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.872205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.871635 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-socket\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.872205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.871710 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-certs\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.872205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.871769 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-credential-socket\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.873959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.873931 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-envoy\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.874130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.874110 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-podinfo\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.884107 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.884055 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-token\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:34.885427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:34.885396 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dnbm\" (UniqueName: \"kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-kube-api-access-5dnbm\") pod \"e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:35.083194 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:35.083112 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:35.391061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:35.391016 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns"] Oct 31 13:35:35.404110 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:35.404067 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:35:35.404218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:35.404159 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:35:35.404218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:35.404197 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:35:35.868621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:35.858863 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" event={"ID":"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732","Type":"ContainerStarted","Data":"0927e77e1dc6ff013793181212c82b0b1d6ec76025504df6f05368c223e76448"} Oct 31 13:35:35.868621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:35.858971 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" event={"ID":"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732","Type":"ContainerStarted","Data":"464d793504f8a98622bc68bb880a1c35b741fc1f18a19ae94d30849c4ba2ed76"} Oct 31 13:35:36.087574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:36.086932 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:36.087574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:36.087046 5532 patch_prober.go:29] interesting pod/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:35:36.087574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:36.087088 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 13:35:36.830827 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:36.830761 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" podStartSLOduration=2.830745 podStartE2EDuration="2.830745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:35.902323489 +0000 UTC m=+1904.048067318" watchObservedRunningTime="2026-10-31 13:35:36.830745 +0000 UTC m=+1904.976488831" Oct 31 13:35:36.832932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:36.832910 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr"] Oct 31 13:35:36.838060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:36.838040 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" Oct 31 13:35:36.848947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:36.848923 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-15dc2f\"" Oct 31 13:35:36.853587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:36.853562 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr"] Oct 31 13:35:36.942844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:36.942791 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg2w6\" (UniqueName: \"kubernetes.io/projected/b45f1106-92ed-4bad-b7c0-aa3a0442793b-kube-api-access-tg2w6\") pod \"payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr\" (UID: \"b45f1106-92ed-4bad-b7c0-aa3a0442793b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" Oct 31 13:35:36.943310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:36.942959 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45f1106-92ed-4bad-b7c0-aa3a0442793b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr\" (UID: \"b45f1106-92ed-4bad-b7c0-aa3a0442793b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" Oct 31 13:35:37.053666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:37.052062 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg2w6\" (UniqueName: \"kubernetes.io/projected/b45f1106-92ed-4bad-b7c0-aa3a0442793b-kube-api-access-tg2w6\") pod \"payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr\" (UID: \"b45f1106-92ed-4bad-b7c0-aa3a0442793b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" Oct 31 13:35:37.053666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:37.052263 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45f1106-92ed-4bad-b7c0-aa3a0442793b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr\" (UID: \"b45f1106-92ed-4bad-b7c0-aa3a0442793b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" Oct 31 13:35:37.053666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:37.053425 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45f1106-92ed-4bad-b7c0-aa3a0442793b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr\" (UID: \"b45f1106-92ed-4bad-b7c0-aa3a0442793b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" Oct 31 13:35:37.078767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:37.078724 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg2w6\" (UniqueName: \"kubernetes.io/projected/b45f1106-92ed-4bad-b7c0-aa3a0442793b-kube-api-access-tg2w6\") pod \"payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr\" (UID: \"b45f1106-92ed-4bad-b7c0-aa3a0442793b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" Oct 31 13:35:37.092556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:37.092473 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:37.157618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:37.157025 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" Oct 31 13:35:37.538009 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:35:37.537978 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45f1106_92ed_4bad_b7c0_aa3a0442793b.slice/crio-6c398d001f3eeca308e58e0d56af841998d14c920558e1c75ddcc1edf9a447ab WatchSource:0}: Error finding container 6c398d001f3eeca308e58e0d56af841998d14c920558e1c75ddcc1edf9a447ab: Status 404 returned error can't find the container with id 6c398d001f3eeca308e58e0d56af841998d14c920558e1c75ddcc1edf9a447ab Oct 31 13:35:37.539744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:37.539718 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr"] Oct 31 13:35:37.885281 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:37.885191 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" event={"ID":"b45f1106-92ed-4bad-b7c0-aa3a0442793b","Type":"ContainerStarted","Data":"011936b68cb4fbbd96119431b3062007e1fda2275ecf43592ca1381f6c33f015"} Oct 31 13:35:37.885281 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:37.885253 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" event={"ID":"b45f1106-92ed-4bad-b7c0-aa3a0442793b","Type":"ContainerStarted","Data":"6c398d001f3eeca308e58e0d56af841998d14c920558e1c75ddcc1edf9a447ab"} Oct 31 13:35:37.885501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:37.885286 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" Oct 31 13:35:37.885677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:37.885653 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:37.886770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:37.886745 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:35:37.906309 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:37.906250 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" podStartSLOduration=1.906231429 podStartE2EDuration="1.906231429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:37.90472228 +0000 UTC m=+1906.050466110" watchObservedRunningTime="2026-10-31 13:35:37.906231429 +0000 UTC m=+1906.051975258" Oct 31 13:35:39.720570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:39.720534 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d"] Oct 31 13:35:39.724772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:39.724754 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" Oct 31 13:35:39.732699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:39.732672 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cbb3d5-dockercfg-vxxvs\"" Oct 31 13:35:39.732934 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:39.732912 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cbb3d5\"" Oct 31 13:35:39.735647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:39.735622 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d"] Oct 31 13:35:39.807974 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:39.807939 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppd9j\" (UniqueName: \"kubernetes.io/projected/ef722ded-585d-4913-9691-aa5170f51a15-kube-api-access-ppd9j\") pod \"payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d\" (UID: \"ef722ded-585d-4913-9691-aa5170f51a15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" Oct 31 13:35:39.808176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:39.808071 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef722ded-585d-4913-9691-aa5170f51a15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d\" (UID: \"ef722ded-585d-4913-9691-aa5170f51a15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" Oct 31 13:35:39.921618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:39.917120 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppd9j\" (UniqueName: \"kubernetes.io/projected/ef722ded-585d-4913-9691-aa5170f51a15-kube-api-access-ppd9j\") pod \"payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d\" (UID: \"ef722ded-585d-4913-9691-aa5170f51a15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" Oct 31 13:35:39.921618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:39.917248 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef722ded-585d-4913-9691-aa5170f51a15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d\" (UID: \"ef722ded-585d-4913-9691-aa5170f51a15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" Oct 31 13:35:39.921618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:39.918669 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef722ded-585d-4913-9691-aa5170f51a15-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d\" (UID: \"ef722ded-585d-4913-9691-aa5170f51a15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" Oct 31 13:35:39.970679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:39.969116 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppd9j\" (UniqueName: \"kubernetes.io/projected/ef722ded-585d-4913-9691-aa5170f51a15-kube-api-access-ppd9j\") pod \"payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d\" (UID: \"ef722ded-585d-4913-9691-aa5170f51a15\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" Oct 31 13:35:40.045720 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:40.043548 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" Oct 31 13:35:40.372460 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:35:40.372426 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef722ded_585d_4913_9691_aa5170f51a15.slice/crio-6ff9c0ce82a6a1ae0626510f14ece48be761de069fde04e9c96f8050ba43a78a WatchSource:0}: Error finding container 6ff9c0ce82a6a1ae0626510f14ece48be761de069fde04e9c96f8050ba43a78a: Status 404 returned error can't find the container with id 6ff9c0ce82a6a1ae0626510f14ece48be761de069fde04e9c96f8050ba43a78a Oct 31 13:35:40.372783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:40.372759 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d"] Oct 31 13:35:40.984356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:40.984314 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" event={"ID":"ef722ded-585d-4913-9691-aa5170f51a15","Type":"ContainerStarted","Data":"5c1a0a3e6ea673efc6bc67b0cac153a0c8ab0e9b530d546d337fb9cbcaa69b1c"} Oct 31 13:35:40.984849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:40.984381 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" event={"ID":"ef722ded-585d-4913-9691-aa5170f51a15","Type":"ContainerStarted","Data":"6ff9c0ce82a6a1ae0626510f14ece48be761de069fde04e9c96f8050ba43a78a"} Oct 31 13:35:40.984849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:40.984413 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" Oct 31 13:35:41.004146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:41.004091 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" podStartSLOduration=2.004077018 podStartE2EDuration="2.004077018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:41.002937002 +0000 UTC m=+1909.148680832" watchObservedRunningTime="2026-10-31 13:35:41.004077018 +0000 UTC m=+1909.149820862" Oct 31 13:35:43.982824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:43.982781 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg"] Oct 31 13:35:43.983284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:43.983109 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerName="istio-proxy" containerID="cri-o://3dd8f197f0c013999e31377a1d078d2d0330ee1d92ae773e2e941bcad99daf30" gracePeriod=30 Oct 31 13:35:48.033098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:48.033064 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr"] Oct 31 13:35:48.037085 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:48.037057 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1447791b-lh9jr"] Oct 31 13:35:48.908034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:48.908004 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" Oct 31 13:35:49.278766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.278740 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:35:49.412553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.412511 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-envoy\") pod \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " Oct 31 13:35:49.412747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.412634 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-socket\") pod \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " Oct 31 13:35:49.412828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.412787 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-certs\") pod \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " Oct 31 13:35:49.412878 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.412830 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-data\") pod \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " Oct 31 13:35:49.412953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.412930 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-socket" pod "83821ff1-96c3-41a0-a3e7-4940a2c1bb76" (UID: "83821ff1-96c3-41a0-a3e7-4940a2c1bb76"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:49.412953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.412945 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istiod-ca-cert\") pod \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " Oct 31 13:35:49.413084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.413006 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-credential-socket\") pod \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " Oct 31 13:35:49.413084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.413036 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-certs" pod "83821ff1-96c3-41a0-a3e7-4940a2c1bb76" (UID: "83821ff1-96c3-41a0-a3e7-4940a2c1bb76"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:49.413196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.413099 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-podinfo\") pod \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " Oct 31 13:35:49.413196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.413127 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-data" pod "83821ff1-96c3-41a0-a3e7-4940a2c1bb76" (UID: "83821ff1-96c3-41a0-a3e7-4940a2c1bb76"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:49.413305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.413208 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-token\" (UniqueName: \"kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-token\") pod \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " Oct 31 13:35:49.413305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.413213 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-credential-socket" pod "83821ff1-96c3-41a0-a3e7-4940a2c1bb76" (UID: "83821ff1-96c3-41a0-a3e7-4940a2c1bb76"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:49.413305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.413246 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-kube-api-access-sgsmc\" (UniqueName: \"kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-kube-api-access-sgsmc\") pod \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\" (UID: \"83821ff1-96c3-41a0-a3e7-4940a2c1bb76\") " Oct 31 13:35:49.413454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.413336 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istiod-ca-cert" pod "83821ff1-96c3-41a0-a3e7-4940a2c1bb76" (UID: "83821ff1-96c3-41a0-a3e7-4940a2c1bb76"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:35:49.414001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.413945 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:49.414001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.413964 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:49.414001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.413974 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:49.414001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.413983 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:49.414001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.413993 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:49.415231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.415198 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-envoy" pod "83821ff1-96c3-41a0-a3e7-4940a2c1bb76" (UID: "83821ff1-96c3-41a0-a3e7-4940a2c1bb76"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:35:49.415358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.415339 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-token" pod "83821ff1-96c3-41a0-a3e7-4940a2c1bb76" (UID: "83821ff1-96c3-41a0-a3e7-4940a2c1bb76"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:49.415438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.415340 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-podinfo" pod "83821ff1-96c3-41a0-a3e7-4940a2c1bb76" (UID: "83821ff1-96c3-41a0-a3e7-4940a2c1bb76"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:35:49.415634 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.415612 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-kube-api-access-sgsmc" pod "83821ff1-96c3-41a0-a3e7-4940a2c1bb76" (UID: "83821ff1-96c3-41a0-a3e7-4940a2c1bb76"). InnerVolumeSpecName "kube-api-access-sgsmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:35:49.516393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.516355 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:49.516586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.516401 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgsmc\" (UniqueName: \"kubernetes.io/projected/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-kube-api-access-sgsmc\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:49.516586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.516416 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:49.516586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:49.516428 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83821ff1-96c3-41a0-a3e7-4940a2c1bb76-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:35:50.006167 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:50.006135 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdde2ce8-b77f-48c8-ab0a-808aab30216c" path="/var/lib/kubelet/pods/cdde2ce8-b77f-48c8-ab0a-808aab30216c/volumes" Oct 31 13:35:50.092832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:50.092798 5532 generic.go:356] "Generic (PLEG): container finished" podID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerID="3dd8f197f0c013999e31377a1d078d2d0330ee1d92ae773e2e941bcad99daf30" exitCode=0 Oct 31 13:35:50.093025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:50.092853 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" Oct 31 13:35:50.093025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:50.092885 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" event={"ID":"83821ff1-96c3-41a0-a3e7-4940a2c1bb76","Type":"ContainerDied","Data":"3dd8f197f0c013999e31377a1d078d2d0330ee1d92ae773e2e941bcad99daf30"} Oct 31 13:35:50.093025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:50.092941 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg" event={"ID":"83821ff1-96c3-41a0-a3e7-4940a2c1bb76","Type":"ContainerDied","Data":"07eaf07179a26e89e6e856b407a60da9942cc2b4bc6567a9033321e6a48fb824"} Oct 31 13:35:50.093025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:50.092965 5532 scope.go:122] "RemoveContainer" containerID="3dd8f197f0c013999e31377a1d078d2d0330ee1d92ae773e2e941bcad99daf30" Oct 31 13:35:50.104255 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:50.104235 5532 scope.go:122] "RemoveContainer" containerID="c8417332ecee36c7a2c0bec68c7fb900f3024294fa5735e5ad97b477ab5aa67f" Oct 31 13:35:50.114867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:50.114669 5532 scope.go:122] "RemoveContainer" containerID="3dd8f197f0c013999e31377a1d078d2d0330ee1d92ae773e2e941bcad99daf30" Oct 31 13:35:50.114982 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:35:50.114962 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dd8f197f0c013999e31377a1d078d2d0330ee1d92ae773e2e941bcad99daf30\": container with ID starting with 3dd8f197f0c013999e31377a1d078d2d0330ee1d92ae773e2e941bcad99daf30 not found: ID does not exist" containerID="3dd8f197f0c013999e31377a1d078d2d0330ee1d92ae773e2e941bcad99daf30" Oct 31 13:35:50.115028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:50.114993 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dd8f197f0c013999e31377a1d078d2d0330ee1d92ae773e2e941bcad99daf30"} err="failed to get container status \"3dd8f197f0c013999e31377a1d078d2d0330ee1d92ae773e2e941bcad99daf30\": rpc error: code = NotFound desc = could not find container \"3dd8f197f0c013999e31377a1d078d2d0330ee1d92ae773e2e941bcad99daf30\": container with ID starting with 3dd8f197f0c013999e31377a1d078d2d0330ee1d92ae773e2e941bcad99daf30 not found: ID does not exist" Oct 31 13:35:50.115028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:50.115013 5532 scope.go:122] "RemoveContainer" containerID="c8417332ecee36c7a2c0bec68c7fb900f3024294fa5735e5ad97b477ab5aa67f" Oct 31 13:35:50.115289 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:35:50.115268 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8417332ecee36c7a2c0bec68c7fb900f3024294fa5735e5ad97b477ab5aa67f\": container with ID starting with c8417332ecee36c7a2c0bec68c7fb900f3024294fa5735e5ad97b477ab5aa67f not found: ID does not exist" containerID="c8417332ecee36c7a2c0bec68c7fb900f3024294fa5735e5ad97b477ab5aa67f" Oct 31 13:35:50.115339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:50.115296 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8417332ecee36c7a2c0bec68c7fb900f3024294fa5735e5ad97b477ab5aa67f"} err="failed to get container status \"c8417332ecee36c7a2c0bec68c7fb900f3024294fa5735e5ad97b477ab5aa67f\": rpc error: code = NotFound desc = could not find container \"c8417332ecee36c7a2c0bec68c7fb900f3024294fa5735e5ad97b477ab5aa67f\": container with ID starting with c8417332ecee36c7a2c0bec68c7fb900f3024294fa5735e5ad97b477ab5aa67f not found: ID does not exist" Oct 31 13:35:50.122917 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:50.122896 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg"] Oct 31 13:35:50.126964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:50.126942 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-466002-openshift-default-6b4d4c696f-f9msg"] Oct 31 13:35:52.009550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:52.009518 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" path="/var/lib/kubelet/pods/83821ff1-96c3-41a0-a3e7-4940a2c1bb76/volumes" Oct 31 13:35:52.020722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:52.020695 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" Oct 31 13:35:53.125057 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:53.125021 5532 scope.go:122] "RemoveContainer" containerID="f340ad496f66f9008e523a38bbb6f658690dd3bbd868952ac087c86acf8a00d7" Oct 31 13:35:53.149069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:53.149044 5532 scope.go:122] "RemoveContainer" containerID="c370d6987978092af6f5c18f9c7af99c9a3160add1de3a70177e46801461732d" Oct 31 13:35:53.160716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:53.160690 5532 scope.go:122] "RemoveContainer" containerID="68dc0210f892b713cb3a14c7f6da1d02b53a5684047822a51e4b2453e6675785" Oct 31 13:35:53.178148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:53.178116 5532 scope.go:122] "RemoveContainer" containerID="d0000a8cf098e0d9596a57e8c74f96722c6bdcbebaa4b427dbeb13cd9ca97bfd" Oct 31 13:35:53.188737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:53.188721 5532 scope.go:122] "RemoveContainer" containerID="652d863d57a8df9596029ce32c67be6a9ed6d883f4e5fb0e6765db00a6a6e485" Oct 31 13:35:53.212898 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:53.212874 5532 scope.go:122] "RemoveContainer" containerID="a4e955e7acb7a86600bc3c64888d0f816aa3c6f8bbb9a23346fed17f6e57fb88" Oct 31 13:35:57.492052 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.492020 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p"] Oct 31 13:35:57.492897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.492875 5532 cpu_manager.go:414] "containerMap: removing container" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerName="istio-proxy" Oct 31 13:35:57.492897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.492895 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerName="istio-proxy" Oct 31 13:35:57.493075 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.492936 5532 cpu_manager.go:414] "containerMap: removing container" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerName="istio-proxy" Oct 31 13:35:57.493075 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.492942 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerName="istio-proxy" Oct 31 13:35:57.493280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.493077 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerName="istio-proxy" Oct 31 13:35:57.499635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.499576 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:57.526054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.526021 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cbb3d5-serving-cert\"" Oct 31 13:35:57.535963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.532859 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p"] Oct 31 13:35:57.609357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.609323 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8305b055-0baa-438e-b283-c4aca7c0926b-openshift-service-ca\") pod \"maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:57.609357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.609363 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhbzh\" (UniqueName: \"kubernetes.io/projected/8305b055-0baa-438e-b283-c4aca7c0926b-kube-api-access-lhbzh\") pod \"maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:57.609665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.609638 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-metrics-certs\") pod \"maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:57.609819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.609791 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-maas-api-tls\") pod \"maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:57.754986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.737262 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8305b055-0baa-438e-b283-c4aca7c0926b-openshift-service-ca\") pod \"maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:57.755287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.755253 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhbzh\" (UniqueName: \"kubernetes.io/projected/8305b055-0baa-438e-b283-c4aca7c0926b-kube-api-access-lhbzh\") pod \"maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:57.755859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.755824 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-metrics-certs\") pod \"maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:57.756086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.739523 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8305b055-0baa-438e-b283-c4aca7c0926b-openshift-service-ca\") pod \"maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:57.756230 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.756212 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-maas-api-tls\") pod \"maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:57.797611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.797555 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-maas-api-tls\") pod \"maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:57.800856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.800826 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhbzh\" (UniqueName: \"kubernetes.io/projected/8305b055-0baa-438e-b283-c4aca7c0926b-kube-api-access-lhbzh\") pod \"maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:57.810773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.810736 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-metrics-certs\") pod \"maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:57.837047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:57.837012 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:58.159130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:58.159100 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p"] Oct 31 13:35:58.160018 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:35:58.159989 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8305b055_0baa_438e_b283_c4aca7c0926b.slice/crio-67d38ef7112685a66a41d7e6ccf2ade7ebbfc45a7fb3a1aa5b3c12a33a7aa8a3 WatchSource:0}: Error finding container 67d38ef7112685a66a41d7e6ccf2ade7ebbfc45a7fb3a1aa5b3c12a33a7aa8a3: Status 404 returned error can't find the container with id 67d38ef7112685a66a41d7e6ccf2ade7ebbfc45a7fb3a1aa5b3c12a33a7aa8a3 Oct 31 13:35:58.282484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:58.282440 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" event={"ID":"8305b055-0baa-438e-b283-c4aca7c0926b","Type":"ContainerStarted","Data":"67d38ef7112685a66a41d7e6ccf2ade7ebbfc45a7fb3a1aa5b3c12a33a7aa8a3"} Oct 31 13:35:58.677396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:58.677363 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h"] Oct 31 13:35:59.292588 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:59.292548 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" event={"ID":"8305b055-0baa-438e-b283-c4aca7c0926b","Type":"ContainerStarted","Data":"6cb755eb84e54fbf57fb94a451884ebde91277b7231a9317a4a91b4fc5b1f61d"} Oct 31 13:35:59.292914 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:59.292637 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:35:59.318744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:35:59.318677 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" podStartSLOduration=1.799987172 podStartE2EDuration="2.318657637s" totalImagesPullingTime="518.670465ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:35:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:35:59.314966145 +0000 UTC m=+1927.460709977" watchObservedRunningTime="2026-10-31 13:35:59.318657637 +0000 UTC m=+1927.464401467" Oct 31 13:36:00.211838 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.211799 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67"] Oct 31 13:36:00.212375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.212051 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" podUID="692f58c7-b8b9-4ca5-aedc-8f4a74804549" containerName="maas-api" containerID="cri-o://0f2f8a327b0117b9e7c76d7ee02c6a0df143e0a4616f363c9571c3d869d4c260" gracePeriod=30 Oct 31 13:36:00.552376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.552342 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:36:00.663236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.663200 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/692f58c7-b8b9-4ca5-aedc-8f4a74804549-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/692f58c7-b8b9-4ca5-aedc-8f4a74804549-openshift-service-ca\") pod \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " Oct 31 13:36:00.663485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.663466 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-metrics-certs\" (UniqueName: \"kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-metrics-certs\") pod \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " Oct 31 13:36:00.663560 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.663496 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/692f58c7-b8b9-4ca5-aedc-8f4a74804549-openshift-service-ca" pod "692f58c7-b8b9-4ca5-aedc-8f4a74804549" (UID: "692f58c7-b8b9-4ca5-aedc-8f4a74804549"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:36:00.663560 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.663514 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-maas-api-tls\") pod \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " Oct 31 13:36:00.663687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.663655 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/692f58c7-b8b9-4ca5-aedc-8f4a74804549-kube-api-access-6g7zr\" (UniqueName: \"kubernetes.io/projected/692f58c7-b8b9-4ca5-aedc-8f4a74804549-kube-api-access-6g7zr\") pod \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\" (UID: \"692f58c7-b8b9-4ca5-aedc-8f4a74804549\") " Oct 31 13:36:00.664357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.664336 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/692f58c7-b8b9-4ca5-aedc-8f4a74804549-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:00.665577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.665546 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-maas-api-tls" pod "692f58c7-b8b9-4ca5-aedc-8f4a74804549" (UID: "692f58c7-b8b9-4ca5-aedc-8f4a74804549"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:36:00.665577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.665566 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-metrics-certs" pod "692f58c7-b8b9-4ca5-aedc-8f4a74804549" (UID: "692f58c7-b8b9-4ca5-aedc-8f4a74804549"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:36:00.665738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.665705 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/692f58c7-b8b9-4ca5-aedc-8f4a74804549-kube-api-access-6g7zr" pod "692f58c7-b8b9-4ca5-aedc-8f4a74804549" (UID: "692f58c7-b8b9-4ca5-aedc-8f4a74804549"). InnerVolumeSpecName "kube-api-access-6g7zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:00.775278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.775180 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:00.775522 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.775505 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/692f58c7-b8b9-4ca5-aedc-8f4a74804549-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:00.775648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:00.775634 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g7zr\" (UniqueName: \"kubernetes.io/projected/692f58c7-b8b9-4ca5-aedc-8f4a74804549-kube-api-access-6g7zr\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:01.317161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:01.317125 5532 generic.go:356] "Generic (PLEG): container finished" podID="692f58c7-b8b9-4ca5-aedc-8f4a74804549" containerID="0f2f8a327b0117b9e7c76d7ee02c6a0df143e0a4616f363c9571c3d869d4c260" exitCode=0 Oct 31 13:36:01.317663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:01.317171 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" Oct 31 13:36:01.317663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:01.317205 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" event={"ID":"692f58c7-b8b9-4ca5-aedc-8f4a74804549","Type":"ContainerDied","Data":"0f2f8a327b0117b9e7c76d7ee02c6a0df143e0a4616f363c9571c3d869d4c260"} Oct 31 13:36:01.317663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:01.317275 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67" event={"ID":"692f58c7-b8b9-4ca5-aedc-8f4a74804549","Type":"ContainerDied","Data":"c8ce57bbd86d9bc9ad08d69676441b9467cdc555bed24ee1c2249326483434b5"} Oct 31 13:36:01.317663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:01.317310 5532 scope.go:122] "RemoveContainer" containerID="0f2f8a327b0117b9e7c76d7ee02c6a0df143e0a4616f363c9571c3d869d4c260" Oct 31 13:36:01.330420 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:01.330401 5532 scope.go:122] "RemoveContainer" containerID="0f2f8a327b0117b9e7c76d7ee02c6a0df143e0a4616f363c9571c3d869d4c260" Oct 31 13:36:01.331572 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:36:01.331542 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f2f8a327b0117b9e7c76d7ee02c6a0df143e0a4616f363c9571c3d869d4c260\": container with ID starting with 0f2f8a327b0117b9e7c76d7ee02c6a0df143e0a4616f363c9571c3d869d4c260 not found: ID does not exist" containerID="0f2f8a327b0117b9e7c76d7ee02c6a0df143e0a4616f363c9571c3d869d4c260" Oct 31 13:36:01.331721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:01.331580 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f2f8a327b0117b9e7c76d7ee02c6a0df143e0a4616f363c9571c3d869d4c260"} err="failed to get container status \"0f2f8a327b0117b9e7c76d7ee02c6a0df143e0a4616f363c9571c3d869d4c260\": rpc error: code = NotFound desc = could not find container \"0f2f8a327b0117b9e7c76d7ee02c6a0df143e0a4616f363c9571c3d869d4c260\": container with ID starting with 0f2f8a327b0117b9e7c76d7ee02c6a0df143e0a4616f363c9571c3d869d4c260 not found: ID does not exist" Oct 31 13:36:01.349648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:01.347872 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67"] Oct 31 13:36:01.361545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:01.361504 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-79f4fb0f-7b866d6df4-kcp67"] Oct 31 13:36:01.773898 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:36:01.773835 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:36:02.005725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:02.005692 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="692f58c7-b8b9-4ca5-aedc-8f4a74804549" path="/var/lib/kubelet/pods/692f58c7-b8b9-4ca5-aedc-8f4a74804549/volumes" Oct 31 13:36:03.677959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:03.677906 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" podUID="70bec635-576b-4a02-999e-c43a064633b0" containerName="payload-processing" containerID="cri-o://9219fffc1b70e1390dafed4aefd2e0210697f5c636b768c2df3d7dcc7fdaf490" gracePeriod=25 Oct 31 13:36:04.006954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.006928 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" Oct 31 13:36:04.149538 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.149500 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70bec635-576b-4a02-999e-c43a064633b0-kube-api-access-8qqfr\" (UniqueName: \"kubernetes.io/projected/70bec635-576b-4a02-999e-c43a064633b0-kube-api-access-8qqfr\") pod \"70bec635-576b-4a02-999e-c43a064633b0\" (UID: \"70bec635-576b-4a02-999e-c43a064633b0\") " Oct 31 13:36:04.149793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.149576 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70bec635-576b-4a02-999e-c43a064633b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bec635-576b-4a02-999e-c43a064633b0-plugins-config-volume\") pod \"70bec635-576b-4a02-999e-c43a064633b0\" (UID: \"70bec635-576b-4a02-999e-c43a064633b0\") " Oct 31 13:36:04.150031 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.149997 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70bec635-576b-4a02-999e-c43a064633b0-plugins-config-volume" pod "70bec635-576b-4a02-999e-c43a064633b0" (UID: "70bec635-576b-4a02-999e-c43a064633b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:36:04.150305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.150286 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70bec635-576b-4a02-999e-c43a064633b0-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:04.151704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.151678 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70bec635-576b-4a02-999e-c43a064633b0-kube-api-access-8qqfr" pod "70bec635-576b-4a02-999e-c43a064633b0" (UID: "70bec635-576b-4a02-999e-c43a064633b0"). InnerVolumeSpecName "kube-api-access-8qqfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:04.252033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.251955 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qqfr\" (UniqueName: \"kubernetes.io/projected/70bec635-576b-4a02-999e-c43a064633b0-kube-api-access-8qqfr\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:04.354279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.354239 5532 generic.go:356] "Generic (PLEG): container finished" podID="70bec635-576b-4a02-999e-c43a064633b0" containerID="9219fffc1b70e1390dafed4aefd2e0210697f5c636b768c2df3d7dcc7fdaf490" exitCode=0 Oct 31 13:36:04.354456 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.354290 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" Oct 31 13:36:04.354456 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.354327 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" event={"ID":"70bec635-576b-4a02-999e-c43a064633b0","Type":"ContainerDied","Data":"9219fffc1b70e1390dafed4aefd2e0210697f5c636b768c2df3d7dcc7fdaf490"} Oct 31 13:36:04.354456 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.354386 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h" event={"ID":"70bec635-576b-4a02-999e-c43a064633b0","Type":"ContainerDied","Data":"09a9ec8833b2f439cc98fde3c0b733fe942a29194510b0e529419f8ebfb2e872"} Oct 31 13:36:04.354456 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.354413 5532 scope.go:122] "RemoveContainer" containerID="9219fffc1b70e1390dafed4aefd2e0210697f5c636b768c2df3d7dcc7fdaf490" Oct 31 13:36:04.366995 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.366976 5532 scope.go:122] "RemoveContainer" containerID="9219fffc1b70e1390dafed4aefd2e0210697f5c636b768c2df3d7dcc7fdaf490" Oct 31 13:36:04.367289 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:36:04.367267 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9219fffc1b70e1390dafed4aefd2e0210697f5c636b768c2df3d7dcc7fdaf490\": container with ID starting with 9219fffc1b70e1390dafed4aefd2e0210697f5c636b768c2df3d7dcc7fdaf490 not found: ID does not exist" containerID="9219fffc1b70e1390dafed4aefd2e0210697f5c636b768c2df3d7dcc7fdaf490" Oct 31 13:36:04.367336 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.367298 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9219fffc1b70e1390dafed4aefd2e0210697f5c636b768c2df3d7dcc7fdaf490"} err="failed to get container status \"9219fffc1b70e1390dafed4aefd2e0210697f5c636b768c2df3d7dcc7fdaf490\": rpc error: code = NotFound desc = could not find container \"9219fffc1b70e1390dafed4aefd2e0210697f5c636b768c2df3d7dcc7fdaf490\": container with ID starting with 9219fffc1b70e1390dafed4aefd2e0210697f5c636b768c2df3d7dcc7fdaf490 not found: ID does not exist" Oct 31 13:36:04.384070 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.384040 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h"] Oct 31 13:36:04.388455 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:04.388427 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-79f4fb0f-78c7766bc9-89b4h"] Oct 31 13:36:05.313704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:05.313674 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:36:06.005998 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:06.005964 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70bec635-576b-4a02-999e-c43a064633b0" path="/var/lib/kubelet/pods/70bec635-576b-4a02-999e-c43a064633b0/volumes" Oct 31 13:36:10.663357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:10.661672 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps"] Oct 31 13:36:10.663357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:10.661882 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" containerID="cri-o://25b19ee1348f14def41c88886f5f3f19776f83768175a4338b4b0819ce388dca" gracePeriod=30 Oct 31 13:36:11.430920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.430879 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb"] Oct 31 13:36:11.431835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.431811 5532 cpu_manager.go:414] "containerMap: removing container" podUID="692f58c7-b8b9-4ca5-aedc-8f4a74804549" containerName="maas-api" Oct 31 13:36:11.431835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.431829 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="692f58c7-b8b9-4ca5-aedc-8f4a74804549" containerName="maas-api" Oct 31 13:36:11.432045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.431888 5532 cpu_manager.go:414] "containerMap: removing container" podUID="70bec635-576b-4a02-999e-c43a064633b0" containerName="payload-processing" Oct 31 13:36:11.432045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.431893 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70bec635-576b-4a02-999e-c43a064633b0" containerName="payload-processing" Oct 31 13:36:11.432045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.432009 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="70bec635-576b-4a02-999e-c43a064633b0" containerName="payload-processing" Oct 31 13:36:11.432045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.432021 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="692f58c7-b8b9-4ca5-aedc-8f4a74804549" containerName="maas-api" Oct 31 13:36:11.432045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.432028 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="83821ff1-96c3-41a0-a3e7-4940a2c1bb76" containerName="istio-proxy" Oct 31 13:36:11.437443 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.437422 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" Oct 31 13:36:11.442333 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.442304 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c45a37\"" Oct 31 13:36:11.447173 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.447021 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb"] Oct 31 13:36:11.576400 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.576360 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brg2w\" (UniqueName: \"kubernetes.io/projected/cec34c9a-e736-4f27-a15d-14532e0e5d6c-kube-api-access-brg2w\") pod \"payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb\" (UID: \"cec34c9a-e736-4f27-a15d-14532e0e5d6c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" Oct 31 13:36:11.586657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.581798 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec34c9a-e736-4f27-a15d-14532e0e5d6c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb\" (UID: \"cec34c9a-e736-4f27-a15d-14532e0e5d6c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" Oct 31 13:36:11.684335 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.684242 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brg2w\" (UniqueName: \"kubernetes.io/projected/cec34c9a-e736-4f27-a15d-14532e0e5d6c-kube-api-access-brg2w\") pod \"payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb\" (UID: \"cec34c9a-e736-4f27-a15d-14532e0e5d6c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" Oct 31 13:36:11.684765 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.684351 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec34c9a-e736-4f27-a15d-14532e0e5d6c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb\" (UID: \"cec34c9a-e736-4f27-a15d-14532e0e5d6c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" Oct 31 13:36:11.685281 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.685249 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec34c9a-e736-4f27-a15d-14532e0e5d6c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb\" (UID: \"cec34c9a-e736-4f27-a15d-14532e0e5d6c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" Oct 31 13:36:11.701716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.701683 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brg2w\" (UniqueName: \"kubernetes.io/projected/cec34c9a-e736-4f27-a15d-14532e0e5d6c-kube-api-access-brg2w\") pod \"payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb\" (UID: \"cec34c9a-e736-4f27-a15d-14532e0e5d6c\") " pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" Oct 31 13:36:11.756291 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:11.756250 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" Oct 31 13:36:12.094155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:12.094131 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb"] Oct 31 13:36:12.095160 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:36:12.095134 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec34c9a_e736_4f27_a15d_14532e0e5d6c.slice/crio-cb1c6d83b02fd03628ab4840ef0b58c681a8875a905f7ccdea86f4a2cb74c14f WatchSource:0}: Error finding container cb1c6d83b02fd03628ab4840ef0b58c681a8875a905f7ccdea86f4a2cb74c14f: Status 404 returned error can't find the container with id cb1c6d83b02fd03628ab4840ef0b58c681a8875a905f7ccdea86f4a2cb74c14f Oct 31 13:36:12.112376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:12.112354 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:36:12.448450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:12.448408 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" event={"ID":"cec34c9a-e736-4f27-a15d-14532e0e5d6c","Type":"ContainerStarted","Data":"c99e3cd4f9498c6df33f3a5cb99bc717739d9a5bcc5999bf840b90323c5b93a0"} Oct 31 13:36:12.448722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:12.448473 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" event={"ID":"cec34c9a-e736-4f27-a15d-14532e0e5d6c","Type":"ContainerStarted","Data":"cb1c6d83b02fd03628ab4840ef0b58c681a8875a905f7ccdea86f4a2cb74c14f"} Oct 31 13:36:12.448722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:12.448620 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" Oct 31 13:36:12.476555 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:12.476493 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" podStartSLOduration=1.476474911 podStartE2EDuration="1.476474911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:12.474005806 +0000 UTC m=+1940.619749634" watchObservedRunningTime="2026-10-31 13:36:12.476474911 +0000 UTC m=+1940.622218740" Oct 31 13:36:13.025152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:13.025114 5532 patch_prober.go:29] interesting pod/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:36:13.025533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:13.025178 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:36:16.025045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.025015 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:36:16.113287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.113256 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-token\" (UniqueName: \"kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-token\") pod \"ceb475d1-1432-4095-b9c1-e6b16f18079a\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " Oct 31 13:36:16.113477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.113311 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-podinfo\") pod \"ceb475d1-1432-4095-b9c1-e6b16f18079a\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " Oct 31 13:36:16.113477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.113459 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-kube-api-access-4v7zx\" (UniqueName: \"kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-kube-api-access-4v7zx\") pod \"ceb475d1-1432-4095-b9c1-e6b16f18079a\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " Oct 31 13:36:16.113656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.113535 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-certs\") pod \"ceb475d1-1432-4095-b9c1-e6b16f18079a\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " Oct 31 13:36:16.113727 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.113655 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceb475d1-1432-4095-b9c1-e6b16f18079a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb475d1-1432-4095-b9c1-e6b16f18079a-istiod-ca-cert\") pod \"ceb475d1-1432-4095-b9c1-e6b16f18079a\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " Oct 31 13:36:16.113819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.113800 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-envoy\") pod \"ceb475d1-1432-4095-b9c1-e6b16f18079a\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " Oct 31 13:36:16.113947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.113931 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-credential-socket\") pod \"ceb475d1-1432-4095-b9c1-e6b16f18079a\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " Oct 31 13:36:16.114028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.113941 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-certs" pod "ceb475d1-1432-4095-b9c1-e6b16f18079a" (UID: "ceb475d1-1432-4095-b9c1-e6b16f18079a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:16.114028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.113972 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-data\") pod \"ceb475d1-1432-4095-b9c1-e6b16f18079a\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " Oct 31 13:36:16.114141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.114080 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-socket\") pod \"ceb475d1-1432-4095-b9c1-e6b16f18079a\" (UID: \"ceb475d1-1432-4095-b9c1-e6b16f18079a\") " Oct 31 13:36:16.115176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.114206 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceb475d1-1432-4095-b9c1-e6b16f18079a-istiod-ca-cert" pod "ceb475d1-1432-4095-b9c1-e6b16f18079a" (UID: "ceb475d1-1432-4095-b9c1-e6b16f18079a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:36:16.115176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.114469 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-credential-socket" pod "ceb475d1-1432-4095-b9c1-e6b16f18079a" (UID: "ceb475d1-1432-4095-b9c1-e6b16f18079a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:16.115176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.114495 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-data" pod "ceb475d1-1432-4095-b9c1-e6b16f18079a" (UID: "ceb475d1-1432-4095-b9c1-e6b16f18079a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:16.115176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.114729 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-socket" pod "ceb475d1-1432-4095-b9c1-e6b16f18079a" (UID: "ceb475d1-1432-4095-b9c1-e6b16f18079a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:16.115176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.115080 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:16.115176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.115103 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ceb475d1-1432-4095-b9c1-e6b16f18079a-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:16.115176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.115119 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:16.115176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.115136 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:16.115176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.115150 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:16.117912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.117645 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-envoy" pod "ceb475d1-1432-4095-b9c1-e6b16f18079a" (UID: "ceb475d1-1432-4095-b9c1-e6b16f18079a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:16.117912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.117655 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-podinfo" pod "ceb475d1-1432-4095-b9c1-e6b16f18079a" (UID: "ceb475d1-1432-4095-b9c1-e6b16f18079a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:36:16.117912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.117653 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-token" pod "ceb475d1-1432-4095-b9c1-e6b16f18079a" (UID: "ceb475d1-1432-4095-b9c1-e6b16f18079a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:16.119123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.119096 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-kube-api-access-4v7zx" pod "ceb475d1-1432-4095-b9c1-e6b16f18079a" (UID: "ceb475d1-1432-4095-b9c1-e6b16f18079a"). InnerVolumeSpecName "kube-api-access-4v7zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:16.216799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.216766 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v7zx\" (UniqueName: \"kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-kube-api-access-4v7zx\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:16.216799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.216795 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:16.216799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.216805 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:16.217028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.216814 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ceb475d1-1432-4095-b9c1-e6b16f18079a-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:16.481969 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.481931 5532 generic.go:356] "Generic (PLEG): container finished" podID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerID="25b19ee1348f14def41c88886f5f3f19776f83768175a4338b4b0819ce388dca" exitCode=0 Oct 31 13:36:16.482153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.481994 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" Oct 31 13:36:16.482153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.482021 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" event={"ID":"ceb475d1-1432-4095-b9c1-e6b16f18079a","Type":"ContainerDied","Data":"25b19ee1348f14def41c88886f5f3f19776f83768175a4338b4b0819ce388dca"} Oct 31 13:36:16.482153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.482084 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps" event={"ID":"ceb475d1-1432-4095-b9c1-e6b16f18079a","Type":"ContainerDied","Data":"40b9b385cf6b90b9481f420dae11527da8da8c3328a05ddb12f1e1324749e3da"} Oct 31 13:36:16.482153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.482110 5532 scope.go:122] "RemoveContainer" containerID="25b19ee1348f14def41c88886f5f3f19776f83768175a4338b4b0819ce388dca" Oct 31 13:36:16.494711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.494691 5532 scope.go:122] "RemoveContainer" containerID="a92b1937e5b263f981c46c09a21fc0f1fc390cbcfac942d4ca2f317a02979f4a" Oct 31 13:36:16.505524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.505504 5532 scope.go:122] "RemoveContainer" containerID="25b19ee1348f14def41c88886f5f3f19776f83768175a4338b4b0819ce388dca" Oct 31 13:36:16.505844 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:36:16.505824 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25b19ee1348f14def41c88886f5f3f19776f83768175a4338b4b0819ce388dca\": container with ID starting with 25b19ee1348f14def41c88886f5f3f19776f83768175a4338b4b0819ce388dca not found: ID does not exist" containerID="25b19ee1348f14def41c88886f5f3f19776f83768175a4338b4b0819ce388dca" Oct 31 13:36:16.505913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.505852 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25b19ee1348f14def41c88886f5f3f19776f83768175a4338b4b0819ce388dca"} err="failed to get container status \"25b19ee1348f14def41c88886f5f3f19776f83768175a4338b4b0819ce388dca\": rpc error: code = NotFound desc = could not find container \"25b19ee1348f14def41c88886f5f3f19776f83768175a4338b4b0819ce388dca\": container with ID starting with 25b19ee1348f14def41c88886f5f3f19776f83768175a4338b4b0819ce388dca not found: ID does not exist" Oct 31 13:36:16.505913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.505869 5532 scope.go:122] "RemoveContainer" containerID="a92b1937e5b263f981c46c09a21fc0f1fc390cbcfac942d4ca2f317a02979f4a" Oct 31 13:36:16.506149 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:36:16.506130 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92b1937e5b263f981c46c09a21fc0f1fc390cbcfac942d4ca2f317a02979f4a\": container with ID starting with a92b1937e5b263f981c46c09a21fc0f1fc390cbcfac942d4ca2f317a02979f4a not found: ID does not exist" containerID="a92b1937e5b263f981c46c09a21fc0f1fc390cbcfac942d4ca2f317a02979f4a" Oct 31 13:36:16.506227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.506156 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92b1937e5b263f981c46c09a21fc0f1fc390cbcfac942d4ca2f317a02979f4a"} err="failed to get container status \"a92b1937e5b263f981c46c09a21fc0f1fc390cbcfac942d4ca2f317a02979f4a\": rpc error: code = NotFound desc = could not find container \"a92b1937e5b263f981c46c09a21fc0f1fc390cbcfac942d4ca2f317a02979f4a\": container with ID starting with a92b1937e5b263f981c46c09a21fc0f1fc390cbcfac942d4ca2f317a02979f4a not found: ID does not exist" Oct 31 13:36:16.520358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.520330 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps"] Oct 31 13:36:16.524873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:16.524841 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-75c956-openshift-default-6995b9c56c-hckps"] Oct 31 13:36:18.006213 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:18.006170 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" path="/var/lib/kubelet/pods/ceb475d1-1432-4095-b9c1-e6b16f18079a/volumes" Oct 31 13:36:21.279466 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:21.279420 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs"] Oct 31 13:36:23.459321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:23.459294 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" Oct 31 13:36:26.280696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.280627 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" podUID="51aea716-c142-4909-a102-5f46f1c68991" containerName="payload-processing" containerID="cri-o://c954889247eb06fdef42e59d0cc0f046a16486f992d1e367131a8739ebcc81d0" gracePeriod=25 Oct 31 13:36:26.299409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.299376 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4"] Oct 31 13:36:26.300698 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.300666 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" Oct 31 13:36:26.300698 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.300687 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" Oct 31 13:36:26.300884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.300726 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" Oct 31 13:36:26.300884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.300732 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" Oct 31 13:36:26.300884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.300879 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" Oct 31 13:36:26.301017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.300892 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceb475d1-1432-4095-b9c1-e6b16f18079a" containerName="istio-proxy" Oct 31 13:36:26.312817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.311115 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.322540 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.322507 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4"] Oct 31 13:36:26.339445 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.339414 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-15dc2f-openshift-default-dockercfg-5tkcp\"" Oct 31 13:36:26.453617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.451936 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.453617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.452014 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.453617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.452115 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.453617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.452246 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.453617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.452277 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.453617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.452388 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ldr2\" (UniqueName: \"kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-kube-api-access-6ldr2\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.453617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.452442 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.453617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.452681 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.453617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.452718 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.556527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.556430 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.556807 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.556773 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.557153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.557133 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.558676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.558614 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.558798 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.558682 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.559163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.559105 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.559712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.557649 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.559933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.559904 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.560047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.560031 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ldr2\" (UniqueName: \"kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-kube-api-access-6ldr2\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.560102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.560088 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.560257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.560240 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.562585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.560275 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.562585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.561654 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.562585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.561869 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.568761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.568734 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.570454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.570407 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.580285 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.580251 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ldr2\" (UniqueName: \"kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-kube-api-access-6ldr2\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.590645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.590567 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.602545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.602518 5532 generic.go:356] "Generic (PLEG): container finished" podID="51aea716-c142-4909-a102-5f46f1c68991" containerID="c954889247eb06fdef42e59d0cc0f046a16486f992d1e367131a8739ebcc81d0" exitCode=0 Oct 31 13:36:26.602738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.602606 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" event={"ID":"51aea716-c142-4909-a102-5f46f1c68991","Type":"ContainerDied","Data":"c954889247eb06fdef42e59d0cc0f046a16486f992d1e367131a8739ebcc81d0"} Oct 31 13:36:26.602738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.602660 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" event={"ID":"51aea716-c142-4909-a102-5f46f1c68991","Type":"ContainerDied","Data":"7a84c633b6422d35bbc48e11f73e1a364a379c11458b26e985f6eef510b4ff21"} Oct 31 13:36:26.602738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.602736 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a84c633b6422d35bbc48e11f73e1a364a379c11458b26e985f6eef510b4ff21" Oct 31 13:36:26.626756 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.626726 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" Oct 31 13:36:26.676207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.676171 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:26.763355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.763320 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51aea716-c142-4909-a102-5f46f1c68991-kube-api-access-v26kf\" (UniqueName: \"kubernetes.io/projected/51aea716-c142-4909-a102-5f46f1c68991-kube-api-access-v26kf\") pod \"51aea716-c142-4909-a102-5f46f1c68991\" (UID: \"51aea716-c142-4909-a102-5f46f1c68991\") " Oct 31 13:36:26.763507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.763446 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51aea716-c142-4909-a102-5f46f1c68991-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51aea716-c142-4909-a102-5f46f1c68991-plugins-config-volume\") pod \"51aea716-c142-4909-a102-5f46f1c68991\" (UID: \"51aea716-c142-4909-a102-5f46f1c68991\") " Oct 31 13:36:26.764139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.764092 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51aea716-c142-4909-a102-5f46f1c68991-plugins-config-volume" pod "51aea716-c142-4909-a102-5f46f1c68991" (UID: "51aea716-c142-4909-a102-5f46f1c68991"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:36:26.764964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.764937 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51aea716-c142-4909-a102-5f46f1c68991-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:26.766586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.766560 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51aea716-c142-4909-a102-5f46f1c68991-kube-api-access-v26kf" pod "51aea716-c142-4909-a102-5f46f1c68991" (UID: "51aea716-c142-4909-a102-5f46f1c68991"). InnerVolumeSpecName "kube-api-access-v26kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:26.867222 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:26.867144 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v26kf\" (UniqueName: \"kubernetes.io/projected/51aea716-c142-4909-a102-5f46f1c68991-kube-api-access-v26kf\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:27.014742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:27.013012 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4"] Oct 31 13:36:27.017144 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:27.017108 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:36:27.017239 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:27.017177 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:36:27.017239 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:27.017205 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:36:27.612822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:27.612713 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" event={"ID":"bfcecb52-1419-43f3-a98d-a3b6f5a624c4","Type":"ContainerStarted","Data":"b8abea8d1e3745a8a9086925568a3e64decf8f308cb2bf5e0960945fc582d68b"} Oct 31 13:36:27.612822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:27.612795 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" event={"ID":"bfcecb52-1419-43f3-a98d-a3b6f5a624c4","Type":"ContainerStarted","Data":"b383b979024be0632524c0e92c9e8140866b27629394fc0ece7f737ed6ec906f"} Oct 31 13:36:27.613357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:27.612857 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs" Oct 31 13:36:27.646982 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:27.646929 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" podStartSLOduration=1.6469158959999999 podStartE2EDuration="1.646915896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:27.64188075 +0000 UTC m=+1955.787624582" watchObservedRunningTime="2026-10-31 13:36:27.646915896 +0000 UTC m=+1955.792659723" Oct 31 13:36:27.676772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:27.676748 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:27.678435 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:27.678401 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs"] Oct 31 13:36:27.683312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:27.683289 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cce3e7-555bb8c665-w28rs"] Oct 31 13:36:28.007308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:28.007275 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51aea716-c142-4909-a102-5f46f1c68991" path="/var/lib/kubelet/pods/51aea716-c142-4909-a102-5f46f1c68991/volumes" Oct 31 13:36:28.705507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:28.705474 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 13:36:28.710534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:28.710496 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" podUID="bfcecb52-1419-43f3-a98d-a3b6f5a624c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 13:36:29.677766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:29.677723 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 13:36:29.677962 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:29.677805 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" podUID="bfcecb52-1419-43f3-a98d-a3b6f5a624c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 13:36:30.676868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:30.676827 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 13:36:30.677354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:30.676885 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" podUID="bfcecb52-1419-43f3-a98d-a3b6f5a624c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 13:36:31.677293 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:31.677260 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 13:36:31.677727 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:31.677317 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" podUID="bfcecb52-1419-43f3-a98d-a3b6f5a624c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 13:36:31.787657 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:36:31.787617 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:36:32.677260 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:32.677222 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 13:36:32.677444 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:32.677282 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" podUID="bfcecb52-1419-43f3-a98d-a3b6f5a624c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 13:36:33.676739 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:33.676706 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" start-of-body= Oct 31 13:36:33.676948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:33.676754 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" podUID="bfcecb52-1419-43f3-a98d-a3b6f5a624c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.156:15021/healthz/ready\": dial tcp 10.132.0.156:15021: connect: connection refused" Oct 31 13:36:34.682213 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:34.682171 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:34.682765 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:34.682741 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:34.683824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:34.683777 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 13:36:36.726068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:36.726033 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4"] Oct 31 13:36:37.016911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.016829 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r"] Oct 31 13:36:37.018501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.018472 5532 cpu_manager.go:414] "containerMap: removing container" podUID="51aea716-c142-4909-a102-5f46f1c68991" containerName="payload-processing" Oct 31 13:36:37.018501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.018499 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51aea716-c142-4909-a102-5f46f1c68991" containerName="payload-processing" Oct 31 13:36:37.018743 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.018701 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="51aea716-c142-4909-a102-5f46f1c68991" containerName="payload-processing" Oct 31 13:36:37.036380 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.036338 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.080040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.079944 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-79f4fb0f-openshift-default-dockercfg-shgv6\"" Oct 31 13:36:37.199894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.199066 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kflrr\" (UniqueName: \"kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-kube-api-access-kflrr\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.199894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.199383 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.199894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.199447 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.199894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.199547 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.199894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.199622 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.199894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.199657 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.199894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.199756 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.199894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.199794 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.204940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.200698 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.228284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.228243 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r"] Oct 31 13:36:37.307107 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.306989 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.312345 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.312312 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.313268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.313247 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.314258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.312967 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.314420 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.314145 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.314572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.314553 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.315063 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.315045 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.315640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.315616 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.315824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.315801 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.316222 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.316198 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.316357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.316282 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.316513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.316502 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kflrr\" (UniqueName: \"kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-kube-api-access-kflrr\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.316585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.316561 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.316654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.315975 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.337373 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.337305 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.349387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.349351 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.349885 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.349857 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kflrr\" (UniqueName: \"kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-kube-api-access-kflrr\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.350768 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.350743 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.527846 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.527808 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:37.830955 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:36:37.830913 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecad2629_f911_440f_ac75_74d7ef9cfdb6.slice/crio-d496011a77a575b4574d49cff6085c9659e62fc79009a4e958a0c1687bc8dd20 WatchSource:0}: Error finding container d496011a77a575b4574d49cff6085c9659e62fc79009a4e958a0c1687bc8dd20: Status 404 returned error can't find the container with id d496011a77a575b4574d49cff6085c9659e62fc79009a4e958a0c1687bc8dd20 Oct 31 13:36:37.831400 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.831118 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r"] Oct 31 13:36:37.834212 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.834165 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:36:37.834333 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.834281 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:36:37.834398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:37.834347 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:36:38.808067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:38.808026 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" event={"ID":"ecad2629-f911-440f-ac75-74d7ef9cfdb6","Type":"ContainerStarted","Data":"2120d0013c76ed07c2a2e3bd2178f813dbc06f3b06b48c736a352035538178b1"} Oct 31 13:36:38.808251 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:38.808088 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" event={"ID":"ecad2629-f911-440f-ac75-74d7ef9cfdb6","Type":"ContainerStarted","Data":"d496011a77a575b4574d49cff6085c9659e62fc79009a4e958a0c1687bc8dd20"} Oct 31 13:36:38.835786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:38.835726 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" podStartSLOduration=2.8357069360000002 podStartE2EDuration="2.835706936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:38.833323628 +0000 UTC m=+1966.979067457" watchObservedRunningTime="2026-10-31 13:36:38.835706936 +0000 UTC m=+1966.981450765" Oct 31 13:36:39.528290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:39.528260 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:39.530558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:39.530527 5532 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:36:39.530735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:39.530585 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" podUID="ecad2629-f911-440f-ac75-74d7ef9cfdb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 13:36:40.528824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:40.528787 5532 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:36:40.529235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:40.528838 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" podUID="ecad2629-f911-440f-ac75-74d7ef9cfdb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 13:36:41.529259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:41.529221 5532 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:36:41.531022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:41.529283 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" podUID="ecad2629-f911-440f-ac75-74d7ef9cfdb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 13:36:41.741623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:41.728517 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" podUID="7e9adade-cc78-4181-b53e-231fd46f9c7f" containerName="payload-processing" containerID="cri-o://e7bcddb2c2696b3b3e52618420d9633acce1403dd18d11d6e1acb2f07feef928" gracePeriod=25 Oct 31 13:36:42.295360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.295060 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" Oct 31 13:36:42.421281 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.421226 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e9adade-cc78-4181-b53e-231fd46f9c7f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9adade-cc78-4181-b53e-231fd46f9c7f-plugins-config-volume\") pod \"7e9adade-cc78-4181-b53e-231fd46f9c7f\" (UID: \"7e9adade-cc78-4181-b53e-231fd46f9c7f\") " Oct 31 13:36:42.421485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.421472 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9adade-cc78-4181-b53e-231fd46f9c7f-kube-api-access-7prwz\" (UniqueName: \"kubernetes.io/projected/7e9adade-cc78-4181-b53e-231fd46f9c7f-kube-api-access-7prwz\") pod \"7e9adade-cc78-4181-b53e-231fd46f9c7f\" (UID: \"7e9adade-cc78-4181-b53e-231fd46f9c7f\") " Oct 31 13:36:42.421736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.421705 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9adade-cc78-4181-b53e-231fd46f9c7f-plugins-config-volume" pod "7e9adade-cc78-4181-b53e-231fd46f9c7f" (UID: "7e9adade-cc78-4181-b53e-231fd46f9c7f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:36:42.422130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.422105 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e9adade-cc78-4181-b53e-231fd46f9c7f-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:42.423548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.423523 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9adade-cc78-4181-b53e-231fd46f9c7f-kube-api-access-7prwz" pod "7e9adade-cc78-4181-b53e-231fd46f9c7f" (UID: "7e9adade-cc78-4181-b53e-231fd46f9c7f"). InnerVolumeSpecName "kube-api-access-7prwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:42.523731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.523582 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7prwz\" (UniqueName: \"kubernetes.io/projected/7e9adade-cc78-4181-b53e-231fd46f9c7f-kube-api-access-7prwz\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:42.535259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.534840 5532 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:36:42.535259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.534891 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" podUID="ecad2629-f911-440f-ac75-74d7ef9cfdb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 13:36:42.917991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.917961 5532 generic.go:356] "Generic (PLEG): container finished" podID="7e9adade-cc78-4181-b53e-231fd46f9c7f" containerID="e7bcddb2c2696b3b3e52618420d9633acce1403dd18d11d6e1acb2f07feef928" exitCode=0 Oct 31 13:36:42.918162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.918007 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" Oct 31 13:36:42.918162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.918050 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" event={"ID":"7e9adade-cc78-4181-b53e-231fd46f9c7f","Type":"ContainerDied","Data":"e7bcddb2c2696b3b3e52618420d9633acce1403dd18d11d6e1acb2f07feef928"} Oct 31 13:36:42.918162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.918106 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4" event={"ID":"7e9adade-cc78-4181-b53e-231fd46f9c7f","Type":"ContainerDied","Data":"3a7fd38c4efd797a54469eb54739bce14ea5902ba60db14f6daff7fbd967eb07"} Oct 31 13:36:42.918162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.918133 5532 scope.go:122] "RemoveContainer" containerID="e7bcddb2c2696b3b3e52618420d9633acce1403dd18d11d6e1acb2f07feef928" Oct 31 13:36:42.930099 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.930079 5532 scope.go:122] "RemoveContainer" containerID="e7bcddb2c2696b3b3e52618420d9633acce1403dd18d11d6e1acb2f07feef928" Oct 31 13:36:42.930402 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:36:42.930382 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7bcddb2c2696b3b3e52618420d9633acce1403dd18d11d6e1acb2f07feef928\": container with ID starting with e7bcddb2c2696b3b3e52618420d9633acce1403dd18d11d6e1acb2f07feef928 not found: ID does not exist" containerID="e7bcddb2c2696b3b3e52618420d9633acce1403dd18d11d6e1acb2f07feef928" Oct 31 13:36:42.930463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.930410 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7bcddb2c2696b3b3e52618420d9633acce1403dd18d11d6e1acb2f07feef928"} err="failed to get container status \"e7bcddb2c2696b3b3e52618420d9633acce1403dd18d11d6e1acb2f07feef928\": rpc error: code = NotFound desc = could not find container \"e7bcddb2c2696b3b3e52618420d9633acce1403dd18d11d6e1acb2f07feef928\": container with ID starting with e7bcddb2c2696b3b3e52618420d9633acce1403dd18d11d6e1acb2f07feef928 not found: ID does not exist" Oct 31 13:36:42.971678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.971644 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4"] Oct 31 13:36:42.976006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:42.975975 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-580f24-86cfcf9c77-ll4l4"] Oct 31 13:36:43.529275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:43.529247 5532 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:36:43.529512 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:43.529488 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" podUID="ecad2629-f911-440f-ac75-74d7ef9cfdb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 13:36:44.010208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:44.010169 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9adade-cc78-4181-b53e-231fd46f9c7f" path="/var/lib/kubelet/pods/7e9adade-cc78-4181-b53e-231fd46f9c7f/volumes" Oct 31 13:36:44.536162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:44.536131 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:44.536714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:44.536687 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:44.537470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:44.537446 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:52.027202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.027167 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd"] Oct 31 13:36:52.028173 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.028153 5532 cpu_manager.go:414] "containerMap: removing container" podUID="7e9adade-cc78-4181-b53e-231fd46f9c7f" containerName="payload-processing" Oct 31 13:36:52.028173 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.028173 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9adade-cc78-4181-b53e-231fd46f9c7f" containerName="payload-processing" Oct 31 13:36:52.028333 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.028323 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9adade-cc78-4181-b53e-231fd46f9c7f" containerName="payload-processing" Oct 31 13:36:52.033423 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.033403 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.051044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.051018 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd"] Oct 31 13:36:52.145247 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.145214 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.145407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.145268 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.145458 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.145418 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkxp8\" (UniqueName: \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-kube-api-access-mkxp8\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.145500 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.145465 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.145500 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.145493 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.145730 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.145713 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.145828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.145805 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.145882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.145869 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.145976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.145961 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.248460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.248397 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.248719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.248493 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.248719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.248657 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.248719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.248713 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.248925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.248821 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkxp8\" (UniqueName: \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-kube-api-access-mkxp8\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.248925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.248876 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.248925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.248916 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.249096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.248957 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.249096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.249058 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.249323 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.249298 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istiod-ca-cert\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.249872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.249815 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-data\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.249872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.249852 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-credential-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.250190 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.250076 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-socket\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.250333 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.250312 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-certs\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.253101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.253061 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-podinfo\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.253296 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.253268 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-envoy\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.263803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.263700 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-token\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.265209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.265179 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkxp8\" (UniqueName: \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-kube-api-access-mkxp8\") pod \"e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.351545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.351465 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:52.645017 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:36:52.644988 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b19b85_cfe2_4365_b9c6_b57dfb53ab24.slice/crio-004896da19461c5258bff6c055335ebfdfe61f8bfa3d039dd5f6a6fc775b83e0 WatchSource:0}: Error finding container 004896da19461c5258bff6c055335ebfdfe61f8bfa3d039dd5f6a6fc775b83e0: Status 404 returned error can't find the container with id 004896da19461c5258bff6c055335ebfdfe61f8bfa3d039dd5f6a6fc775b83e0 Oct 31 13:36:52.645231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.645205 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd"] Oct 31 13:36:52.648802 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.648764 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:36:52.648904 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.648856 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:36:52.648946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:52.648929 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:36:53.020153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.020111 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" event={"ID":"10b19b85-cfe2-4365-b9c6-b57dfb53ab24","Type":"ContainerStarted","Data":"e4b1d96bd5a2226b95c4cf4f911b252c0ffc8a430896f18cb047352eea1792a7"} Oct 31 13:36:53.020359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.020176 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" event={"ID":"10b19b85-cfe2-4365-b9c6-b57dfb53ab24","Type":"ContainerStarted","Data":"004896da19461c5258bff6c055335ebfdfe61f8bfa3d039dd5f6a6fc775b83e0"} Oct 31 13:36:53.045971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.045922 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" podStartSLOduration=1.04590783 podStartE2EDuration="1.04590783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:36:53.045233063 +0000 UTC m=+1981.190976894" watchObservedRunningTime="2026-10-31 13:36:53.04590783 +0000 UTC m=+1981.191651660" Oct 31 13:36:53.319673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.316041 5532 scope.go:122] "RemoveContainer" containerID="0c186ca8d9c1ddb46fd3113a217301f76d290fbecad95da44f05a5dba548bf84" Oct 31 13:36:53.364320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.358671 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:36:53.367572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.367530 5532 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 31 13:36:53.367797 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.367613 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" podUID="10b19b85-cfe2-4365-b9c6-b57dfb53ab24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 31 13:36:53.370413 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.370279 5532 scope.go:122] "RemoveContainer" containerID="2ad008cd68eff71450a8e5b7e1f60c0e603e443efa6f6f1db2d9b63e06c44fb4" Oct 31 13:36:53.382586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.382558 5532 scope.go:122] "RemoveContainer" containerID="85c82577da11c8e35cce26cb614b6fbe4d61378d8b950ad4a7a3914f9dc67287" Oct 31 13:36:53.400063 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.400030 5532 scope.go:122] "RemoveContainer" containerID="54e42553dc1b95a505ec7df6030d5c7b4a9fc04bb60ba7d95c4077fe5b338df0" Oct 31 13:36:53.413146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.413126 5532 scope.go:122] "RemoveContainer" containerID="ec6273d64b0fe913d83a8df7eb74c1550bb5d49ed376e9c131b81b366325c75a" Oct 31 13:36:53.430150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.430127 5532 scope.go:122] "RemoveContainer" containerID="46d8892cd4b13f4e625f5fcefc4675e0483784fba9b8b3c895965300e97309c3" Oct 31 13:36:53.448780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.447347 5532 scope.go:122] "RemoveContainer" containerID="96160a93b2d259704b759b7ad2667625f376515a92028e4ccf1c6bb2db2c4722" Oct 31 13:36:53.473860 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.473838 5532 scope.go:122] "RemoveContainer" containerID="c41b47ac255a29fbc17582cfe760e0de90ab948b2dbd2f6effb89b990261435e" Oct 31 13:36:53.484515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:53.484496 5532 scope.go:122] "RemoveContainer" containerID="d3530bca24c5a18430ab471fa1505520d885e2a04afedaa1b45ac570e60485fd" Oct 31 13:36:54.109615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:54.106119 5532 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" secret="" err="secret \"e2e-mt-79f4fb0f-openshift-default-dockercfg-shgv6\" not found" Oct 31 13:36:54.440196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:54.438941 5532 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 31 13:36:54.440196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:54.439007 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" podUID="10b19b85-cfe2-4365-b9c6-b57dfb53ab24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 31 13:36:54.460075 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:54.452671 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r"] Oct 31 13:36:54.460075 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:54.452747 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd"] Oct 31 13:36:54.460075 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:54.453006 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" podUID="ecad2629-f911-440f-ac75-74d7ef9cfdb6" containerName="istio-proxy" containerID="cri-o://2120d0013c76ed07c2a2e3bd2178f813dbc06f3b06b48c736a352035538178b1" gracePeriod=30 Oct 31 13:36:55.113303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:55.113269 5532 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" secret="" err="secret \"e2e-mt-79f4fb0f-openshift-default-dockercfg-shgv6\" not found" Oct 31 13:36:55.360970 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:55.360931 5532 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 31 13:36:55.361338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:55.361293 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" podUID="10b19b85-cfe2-4365-b9c6-b57dfb53ab24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 31 13:36:56.163708 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:56.162573 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" podUID="10b19b85-cfe2-4365-b9c6-b57dfb53ab24" containerName="istio-proxy" containerID="cri-o://e4b1d96bd5a2226b95c4cf4f911b252c0ffc8a430896f18cb047352eea1792a7" gracePeriod=30 Oct 31 13:36:59.540059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.540017 5532 patch_prober.go:29] interesting pod/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 31 13:36:59.548846 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.548812 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd"] Oct 31 13:36:59.549255 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.549228 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" podUID="6e2aaa6e-c071-4d13-8aa9-857383962fdb" containerName="maas-api" containerID="cri-o://5120043280e5153d95b77b388c43f479868ec905c1c14ac1b80e849ecc5d55a5" gracePeriod=30 Oct 31 13:36:59.551936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.551903 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" podUID="ecad2629-f911-440f-ac75-74d7ef9cfdb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 31 13:36:59.838948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.835960 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:36:59.939408 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.938269 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-envoy\") pod \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " Oct 31 13:36:59.939617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.939406 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-certs" pod "ecad2629-f911-440f-ac75-74d7ef9cfdb6" (UID: "ecad2629-f911-440f-ac75-74d7ef9cfdb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:59.954060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.940337 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-certs\") pod \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " Oct 31 13:36:59.954060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.941789 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-podinfo\") pod \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " Oct 31 13:36:59.954060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.942427 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-envoy" pod "ecad2629-f911-440f-ac75-74d7ef9cfdb6" (UID: "ecad2629-f911-440f-ac75-74d7ef9cfdb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:59.957531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.954983 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-socket\") pod \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " Oct 31 13:36:59.957531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.955230 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-credential-socket\") pod \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " Oct 31 13:36:59.957531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.955348 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-socket" pod "ecad2629-f911-440f-ac75-74d7ef9cfdb6" (UID: "ecad2629-f911-440f-ac75-74d7ef9cfdb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:59.957531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.955669 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-credential-socket" pod "ecad2629-f911-440f-ac75-74d7ef9cfdb6" (UID: "ecad2629-f911-440f-ac75-74d7ef9cfdb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:59.967472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.967435 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-kube-api-access-kflrr\" (UniqueName: \"kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-kube-api-access-kflrr\") pod \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " Oct 31 13:36:59.967712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.967639 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istiod-ca-cert\") pod \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " Oct 31 13:36:59.967712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.967706 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-data\") pod \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " Oct 31 13:36:59.967861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.967804 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-token\" (UniqueName: \"kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-token\") pod \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\" (UID: \"ecad2629-f911-440f-ac75-74d7ef9cfdb6\") " Oct 31 13:36:59.968766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.968726 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:59.968766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.968756 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:59.968766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.968769 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:59.968999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.968781 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:36:59.969851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.969824 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istiod-ca-cert" pod "ecad2629-f911-440f-ac75-74d7ef9cfdb6" (UID: "ecad2629-f911-440f-ac75-74d7ef9cfdb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:36:59.975365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.975328 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-data" pod "ecad2629-f911-440f-ac75-74d7ef9cfdb6" (UID: "ecad2629-f911-440f-ac75-74d7ef9cfdb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:36:59.976348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.976273 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-kube-api-access-kflrr" pod "ecad2629-f911-440f-ac75-74d7ef9cfdb6" (UID: "ecad2629-f911-440f-ac75-74d7ef9cfdb6"). InnerVolumeSpecName "kube-api-access-kflrr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:36:59.999385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:36:59.999263 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-token" pod "ecad2629-f911-440f-ac75-74d7ef9cfdb6" (UID: "ecad2629-f911-440f-ac75-74d7ef9cfdb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:00.000788 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.000413 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:37:00.022615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.021842 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-podinfo" pod "ecad2629-f911-440f-ac75-74d7ef9cfdb6" (UID: "ecad2629-f911-440f-ac75-74d7ef9cfdb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:00.070933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.070825 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kflrr\" (UniqueName: \"kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-kube-api-access-kflrr\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:00.070933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.070864 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:00.070933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.070874 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:00.070933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.070884 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:00.070933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.070892 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ecad2629-f911-440f-ac75-74d7ef9cfdb6-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:00.075946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.075259 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr"] Oct 31 13:37:00.077813 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.077610 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" podUID="48476992-f75d-4caf-aed3-02cd8c339644" containerName="istio-proxy" containerID="cri-o://40899891746f94dbd718f4c325645ca1ebfa0bb51a2ee4435634eb6ba4679847" gracePeriod=30 Oct 31 13:37:00.172422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.172384 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-metrics-certs\") pod \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " Oct 31 13:37:00.172650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.172472 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e2aaa6e-c071-4d13-8aa9-857383962fdb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2aaa6e-c071-4d13-8aa9-857383962fdb-openshift-service-ca\") pod \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " Oct 31 13:37:00.172650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.172518 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e2aaa6e-c071-4d13-8aa9-857383962fdb-kube-api-access-lj6l7\" (UniqueName: \"kubernetes.io/projected/6e2aaa6e-c071-4d13-8aa9-857383962fdb-kube-api-access-lj6l7\") pod \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " Oct 31 13:37:00.172899 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.172880 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-maas-api-tls\") pod \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\" (UID: \"6e2aaa6e-c071-4d13-8aa9-857383962fdb\") " Oct 31 13:37:00.173024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.172882 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e2aaa6e-c071-4d13-8aa9-857383962fdb-openshift-service-ca" pod "6e2aaa6e-c071-4d13-8aa9-857383962fdb" (UID: "6e2aaa6e-c071-4d13-8aa9-857383962fdb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:00.174177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.174147 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e2aaa6e-c071-4d13-8aa9-857383962fdb-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:00.174902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.174876 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-metrics-certs" pod "6e2aaa6e-c071-4d13-8aa9-857383962fdb" (UID: "6e2aaa6e-c071-4d13-8aa9-857383962fdb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:00.175003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.174951 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e2aaa6e-c071-4d13-8aa9-857383962fdb-kube-api-access-lj6l7" pod "6e2aaa6e-c071-4d13-8aa9-857383962fdb" (UID: "6e2aaa6e-c071-4d13-8aa9-857383962fdb"). InnerVolumeSpecName "kube-api-access-lj6l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:00.175043 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.175002 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-maas-api-tls" pod "6e2aaa6e-c071-4d13-8aa9-857383962fdb" (UID: "6e2aaa6e-c071-4d13-8aa9-857383962fdb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:00.317625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.304311 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:00.317625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.304351 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e2aaa6e-c071-4d13-8aa9-857383962fdb-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:00.317625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.304367 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj6l7\" (UniqueName: \"kubernetes.io/projected/6e2aaa6e-c071-4d13-8aa9-857383962fdb-kube-api-access-lj6l7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:00.352666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.349244 5532 generic.go:356] "Generic (PLEG): container finished" podID="6e2aaa6e-c071-4d13-8aa9-857383962fdb" containerID="5120043280e5153d95b77b388c43f479868ec905c1c14ac1b80e849ecc5d55a5" exitCode=0 Oct 31 13:37:00.352666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.349554 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" event={"ID":"6e2aaa6e-c071-4d13-8aa9-857383962fdb","Type":"ContainerDied","Data":"5120043280e5153d95b77b388c43f479868ec905c1c14ac1b80e849ecc5d55a5"} Oct 31 13:37:00.352666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.349634 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" event={"ID":"6e2aaa6e-c071-4d13-8aa9-857383962fdb","Type":"ContainerDied","Data":"cdc93e2a6633bbdb6e40e8121aa35a0401aa889d0673b1a80ec6d50b12308b48"} Oct 31 13:37:00.352666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.349666 5532 scope.go:122] "RemoveContainer" containerID="5120043280e5153d95b77b388c43f479868ec905c1c14ac1b80e849ecc5d55a5" Oct 31 13:37:00.352666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.349830 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd" Oct 31 13:37:00.398665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.398498 5532 generic.go:356] "Generic (PLEG): container finished" podID="ecad2629-f911-440f-ac75-74d7ef9cfdb6" containerID="2120d0013c76ed07c2a2e3bd2178f813dbc06f3b06b48c736a352035538178b1" exitCode=0 Oct 31 13:37:00.398665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.398544 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" event={"ID":"ecad2629-f911-440f-ac75-74d7ef9cfdb6","Type":"ContainerDied","Data":"2120d0013c76ed07c2a2e3bd2178f813dbc06f3b06b48c736a352035538178b1"} Oct 31 13:37:00.398665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.398609 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" event={"ID":"ecad2629-f911-440f-ac75-74d7ef9cfdb6","Type":"ContainerDied","Data":"d496011a77a575b4574d49cff6085c9659e62fc79009a4e958a0c1687bc8dd20"} Oct 31 13:37:00.398972 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.398682 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r" Oct 31 13:37:00.425044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.423852 5532 scope.go:122] "RemoveContainer" containerID="5120043280e5153d95b77b388c43f479868ec905c1c14ac1b80e849ecc5d55a5" Oct 31 13:37:00.426837 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:37:00.426138 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5120043280e5153d95b77b388c43f479868ec905c1c14ac1b80e849ecc5d55a5\": container with ID starting with 5120043280e5153d95b77b388c43f479868ec905c1c14ac1b80e849ecc5d55a5 not found: ID does not exist" containerID="5120043280e5153d95b77b388c43f479868ec905c1c14ac1b80e849ecc5d55a5" Oct 31 13:37:00.426837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.426186 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5120043280e5153d95b77b388c43f479868ec905c1c14ac1b80e849ecc5d55a5"} err="failed to get container status \"5120043280e5153d95b77b388c43f479868ec905c1c14ac1b80e849ecc5d55a5\": rpc error: code = NotFound desc = could not find container \"5120043280e5153d95b77b388c43f479868ec905c1c14ac1b80e849ecc5d55a5\": container with ID starting with 5120043280e5153d95b77b388c43f479868ec905c1c14ac1b80e849ecc5d55a5 not found: ID does not exist" Oct 31 13:37:00.426837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.426223 5532 scope.go:122] "RemoveContainer" containerID="2120d0013c76ed07c2a2e3bd2178f813dbc06f3b06b48c736a352035538178b1" Oct 31 13:37:00.446517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.446481 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r"] Oct 31 13:37:00.458861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.458815 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-q7s8r"] Oct 31 13:37:00.462999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.462977 5532 scope.go:122] "RemoveContainer" containerID="2120d0013c76ed07c2a2e3bd2178f813dbc06f3b06b48c736a352035538178b1" Oct 31 13:37:00.463529 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:37:00.463498 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2120d0013c76ed07c2a2e3bd2178f813dbc06f3b06b48c736a352035538178b1\": container with ID starting with 2120d0013c76ed07c2a2e3bd2178f813dbc06f3b06b48c736a352035538178b1 not found: ID does not exist" containerID="2120d0013c76ed07c2a2e3bd2178f813dbc06f3b06b48c736a352035538178b1" Oct 31 13:37:00.463680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.463661 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2120d0013c76ed07c2a2e3bd2178f813dbc06f3b06b48c736a352035538178b1"} err="failed to get container status \"2120d0013c76ed07c2a2e3bd2178f813dbc06f3b06b48c736a352035538178b1\": rpc error: code = NotFound desc = could not find container \"2120d0013c76ed07c2a2e3bd2178f813dbc06f3b06b48c736a352035538178b1\": container with ID starting with 2120d0013c76ed07c2a2e3bd2178f813dbc06f3b06b48c736a352035538178b1 not found: ID does not exist" Oct 31 13:37:00.477657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.477571 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd"] Oct 31 13:37:00.491163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:00.491118 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cce3e7-764848994d-r7nxd"] Oct 31 13:37:01.764678 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:37:01.764643 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:37:02.235341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:02.235221 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e2aaa6e-c071-4d13-8aa9-857383962fdb" path="/var/lib/kubelet/pods/6e2aaa6e-c071-4d13-8aa9-857383962fdb/volumes" Oct 31 13:37:02.236538 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:02.236511 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecad2629-f911-440f-ac75-74d7ef9cfdb6" path="/var/lib/kubelet/pods/ecad2629-f911-440f-ac75-74d7ef9cfdb6/volumes" Oct 31 13:37:04.595962 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:04.595884 5532 generic.go:356] "Generic (PLEG): container finished" podID="10b19b85-cfe2-4365-b9c6-b57dfb53ab24" containerID="e4b1d96bd5a2226b95c4cf4f911b252c0ffc8a430896f18cb047352eea1792a7" exitCode=0 Oct 31 13:37:04.596435 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:04.596231 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" event={"ID":"10b19b85-cfe2-4365-b9c6-b57dfb53ab24","Type":"ContainerDied","Data":"e4b1d96bd5a2226b95c4cf4f911b252c0ffc8a430896f18cb047352eea1792a7"} Oct 31 13:37:05.381176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.313388 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:37:05.636781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.528925 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/health\": context deadline exceeded" Oct 31 13:37:05.636781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.577638 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:37:05.636781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.579714 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:37:05.636781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.579963 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:37:05.636781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.603504 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:37:05.669985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.669152 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:05.763616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.729675 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:37:05.763616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.748753 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:37:05.789621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.769727 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:05.920530 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.920478 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:05.921418 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.921029 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:37:05.926339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.926079 5532 generic.go:356] "Generic (PLEG): container finished" podID="48476992-f75d-4caf-aed3-02cd8c339644" containerID="40899891746f94dbd718f4c325645ca1ebfa0bb51a2ee4435634eb6ba4679847" exitCode=0 Oct 31 13:37:05.926609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:05.926553 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" event={"ID":"48476992-f75d-4caf-aed3-02cd8c339644","Type":"ContainerDied","Data":"40899891746f94dbd718f4c325645ca1ebfa0bb51a2ee4435634eb6ba4679847"} Oct 31 13:37:06.102262 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.102223 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-socket\") pod \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " Oct 31 13:37:06.105701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.105668 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-token\" (UniqueName: \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-token\") pod \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " Oct 31 13:37:06.115619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.106048 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-data\") pod \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " Oct 31 13:37:06.115941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.115913 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-credential-socket\") pod \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " Oct 31 13:37:06.124768 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.116105 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-kube-api-access-mkxp8\" (UniqueName: \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-kube-api-access-mkxp8\") pod \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " Oct 31 13:37:06.125049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.125029 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-podinfo\") pod \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " Oct 31 13:37:06.125297 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.125277 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-envoy\") pod \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " Oct 31 13:37:06.129622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.125660 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-certs\") pod \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " Oct 31 13:37:06.129813 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.129791 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istiod-ca-cert\") pod \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " Oct 31 13:37:06.167279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.129960 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-data" pod "10b19b85-cfe2-4365-b9c6-b57dfb53ab24" (UID: "10b19b85-cfe2-4365-b9c6-b57dfb53ab24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:06.167864 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.167505 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-credential-socket" pod "10b19b85-cfe2-4365-b9c6-b57dfb53ab24" (UID: "10b19b85-cfe2-4365-b9c6-b57dfb53ab24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:06.168360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.168334 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-socket" pod "10b19b85-cfe2-4365-b9c6-b57dfb53ab24" (UID: "10b19b85-cfe2-4365-b9c6-b57dfb53ab24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:06.168757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.168737 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-certs" pod "10b19b85-cfe2-4365-b9c6-b57dfb53ab24" (UID: "10b19b85-cfe2-4365-b9c6-b57dfb53ab24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:06.169244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.169223 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istiod-ca-cert" pod "10b19b85-cfe2-4365-b9c6-b57dfb53ab24" (UID: "10b19b85-cfe2-4365-b9c6-b57dfb53ab24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:06.270828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.198665 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:06.270828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.198699 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:06.270828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.198717 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:06.270828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.198732 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:06.270828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.198748 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:06.513680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.342692 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:06.513680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.429057 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-envoy" pod "10b19b85-cfe2-4365-b9c6-b57dfb53ab24" (UID: "10b19b85-cfe2-4365-b9c6-b57dfb53ab24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:06.513680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.429265 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:06.788681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.644787 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-token" pod "10b19b85-cfe2-4365-b9c6-b57dfb53ab24" (UID: "10b19b85-cfe2-4365-b9c6-b57dfb53ab24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:06.788681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.645426 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:06.788681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.669361 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:37:06.788681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.669458 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:37:06.788681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.669776 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:37:06.788681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.707205 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:37:06.794034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.790076 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:37:06.861986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.806506 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-podinfo" pod "10b19b85-cfe2-4365-b9c6-b57dfb53ab24" (UID: "10b19b85-cfe2-4365-b9c6-b57dfb53ab24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:06.861986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.806871 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-kube-api-access-mkxp8" pod "10b19b85-cfe2-4365-b9c6-b57dfb53ab24" (UID: "10b19b85-cfe2-4365-b9c6-b57dfb53ab24"). InnerVolumeSpecName "kube-api-access-mkxp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:06.861986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.834290 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:37:06.863302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.863273 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-kube-api-access-mkxp8\" (UniqueName: \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-kube-api-access-mkxp8\") pod \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " Oct 31 13:37:06.865983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:06.865957 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-podinfo\") pod \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " Oct 31 13:37:07.271280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.234873 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-token\" (UniqueName: \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-token\") pod \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\" (UID: \"10b19b85-cfe2-4365-b9c6-b57dfb53ab24\") " Oct 31 13:37:07.271280 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:37:06.863643 5532 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/10b19b85-cfe2-4365-b9c6-b57dfb53ab24/volumes/kubernetes.io~projected/kube-api-access-mkxp8 Oct 31 13:37:07.271280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.240256 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-kube-api-access-mkxp8" pod "10b19b85-cfe2-4365-b9c6-b57dfb53ab24" (UID: "10b19b85-cfe2-4365-b9c6-b57dfb53ab24"). InnerVolumeSpecName "kube-api-access-mkxp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:07.271280 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:37:06.866366 5532 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/10b19b85-cfe2-4365-b9c6-b57dfb53ab24/volumes/kubernetes.io~downward-api/istio-podinfo Oct 31 13:37:07.271280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.240287 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-podinfo" pod "10b19b85-cfe2-4365-b9c6-b57dfb53ab24" (UID: "10b19b85-cfe2-4365-b9c6-b57dfb53ab24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:07.271280 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:37:07.240367 5532 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/10b19b85-cfe2-4365-b9c6-b57dfb53ab24/volumes/kubernetes.io~projected/istio-token Oct 31 13:37:07.271280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.240393 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-token" pod "10b19b85-cfe2-4365-b9c6-b57dfb53ab24" (UID: "10b19b85-cfe2-4365-b9c6-b57dfb53ab24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:07.271280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.241201 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkxp8\" (UniqueName: \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-kube-api-access-mkxp8\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.271280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.241236 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.271280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.241253 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b19b85-cfe2-4365-b9c6-b57dfb53ab24-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.271280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.243938 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:37:07.271280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.247670 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:37:07.294496 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.294467 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:37:07.311817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.311787 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:37:07.377204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.377163 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:37:07.397165 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.396980 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:37:07.423058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.423023 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" Oct 31 13:37:07.424425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.424392 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd" event={"ID":"10b19b85-cfe2-4365-b9c6-b57dfb53ab24","Type":"ContainerDied","Data":"004896da19461c5258bff6c055335ebfdfe61f8bfa3d039dd5f6a6fc775b83e0"} Oct 31 13:37:07.424565 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.424479 5532 scope.go:122] "RemoveContainer" containerID="e4b1d96bd5a2226b95c4cf4f911b252c0ffc8a430896f18cb047352eea1792a7" Oct 31 13:37:07.444712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.444585 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:37:07.617234 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.617202 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd"] Oct 31 13:37:07.621538 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.621512 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-79f4fb0f-openshift-default-85bd9f8bdf-ppwzd"] Oct 31 13:37:07.635293 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.632386 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:37:07.712925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.712894 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-credential-socket\") pod \"48476992-f75d-4caf-aed3-02cd8c339644\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " Oct 31 13:37:07.713199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.713167 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48476992-f75d-4caf-aed3-02cd8c339644-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48476992-f75d-4caf-aed3-02cd8c339644-istio-podinfo\") pod \"48476992-f75d-4caf-aed3-02cd8c339644\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " Oct 31 13:37:07.713317 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.713282 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-data\") pod \"48476992-f75d-4caf-aed3-02cd8c339644\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " Oct 31 13:37:07.713378 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.713315 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-credential-socket" pod "48476992-f75d-4caf-aed3-02cd8c339644" (UID: "48476992-f75d-4caf-aed3-02cd8c339644"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:07.713378 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.713332 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-certs\") pod \"48476992-f75d-4caf-aed3-02cd8c339644\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " Oct 31 13:37:07.713494 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.713429 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-kube-api-access-vnlvc\" (UniqueName: \"kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-kube-api-access-vnlvc\") pod \"48476992-f75d-4caf-aed3-02cd8c339644\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " Oct 31 13:37:07.713578 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.713561 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-data" pod "48476992-f75d-4caf-aed3-02cd8c339644" (UID: "48476992-f75d-4caf-aed3-02cd8c339644"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:07.713578 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.713569 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-socket\") pod \"48476992-f75d-4caf-aed3-02cd8c339644\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " Oct 31 13:37:07.713751 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.713616 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-envoy\") pod \"48476992-f75d-4caf-aed3-02cd8c339644\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " Oct 31 13:37:07.713751 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.713729 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48476992-f75d-4caf-aed3-02cd8c339644-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48476992-f75d-4caf-aed3-02cd8c339644-istiod-ca-cert\") pod \"48476992-f75d-4caf-aed3-02cd8c339644\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " Oct 31 13:37:07.713868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.713780 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-certs" pod "48476992-f75d-4caf-aed3-02cd8c339644" (UID: "48476992-f75d-4caf-aed3-02cd8c339644"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:07.713926 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.713914 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-istio-token\" (UniqueName: \"kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-istio-token\") pod \"48476992-f75d-4caf-aed3-02cd8c339644\" (UID: \"48476992-f75d-4caf-aed3-02cd8c339644\") " Oct 31 13:37:07.713988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.713975 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-socket" pod "48476992-f75d-4caf-aed3-02cd8c339644" (UID: "48476992-f75d-4caf-aed3-02cd8c339644"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:07.715080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.714338 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48476992-f75d-4caf-aed3-02cd8c339644-istiod-ca-cert" pod "48476992-f75d-4caf-aed3-02cd8c339644" (UID: "48476992-f75d-4caf-aed3-02cd8c339644"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:07.715080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.714860 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.715080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.714883 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.715080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.714900 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.715080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.714915 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48476992-f75d-4caf-aed3-02cd8c339644-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.715080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.714929 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.716037 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.716011 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48476992-f75d-4caf-aed3-02cd8c339644-istio-podinfo" pod "48476992-f75d-4caf-aed3-02cd8c339644" (UID: "48476992-f75d-4caf-aed3-02cd8c339644"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:37:07.721271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.721235 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-envoy" pod "48476992-f75d-4caf-aed3-02cd8c339644" (UID: "48476992-f75d-4caf-aed3-02cd8c339644"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:37:07.725893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.725864 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-kube-api-access-vnlvc" pod "48476992-f75d-4caf-aed3-02cd8c339644" (UID: "48476992-f75d-4caf-aed3-02cd8c339644"). InnerVolumeSpecName "kube-api-access-vnlvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:07.726002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.725924 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-istio-token" pod "48476992-f75d-4caf-aed3-02cd8c339644" (UID: "48476992-f75d-4caf-aed3-02cd8c339644"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:07.817006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.816967 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnlvc\" (UniqueName: \"kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-kube-api-access-vnlvc\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.817006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.817009 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48476992-f75d-4caf-aed3-02cd8c339644-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.817424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.817026 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48476992-f75d-4caf-aed3-02cd8c339644-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:07.817424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:07.817041 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48476992-f75d-4caf-aed3-02cd8c339644-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:08.007676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:08.007640 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b19b85-cfe2-4365-b9c6-b57dfb53ab24" path="/var/lib/kubelet/pods/10b19b85-cfe2-4365-b9c6-b57dfb53ab24/volumes" Oct 31 13:37:08.431398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:08.431365 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" Oct 31 13:37:08.431398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:08.431391 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr" event={"ID":"48476992-f75d-4caf-aed3-02cd8c339644","Type":"ContainerDied","Data":"3ffb688c1cf744a20b3a29969f711b581cf29eb3521e4f3530ffcf92d97742d5"} Oct 31 13:37:08.431725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:08.431467 5532 scope.go:122] "RemoveContainer" containerID="40899891746f94dbd718f4c325645ca1ebfa0bb51a2ee4435634eb6ba4679847" Oct 31 13:37:08.488113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:08.488081 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr"] Oct 31 13:37:08.493628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:08.493602 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-zw5sr"] Oct 31 13:37:10.006539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:10.006505 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48476992-f75d-4caf-aed3-02cd8c339644" path="/var/lib/kubelet/pods/48476992-f75d-4caf-aed3-02cd8c339644/volumes" Oct 31 13:37:11.401582 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.401542 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv"] Oct 31 13:37:11.402875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.402849 5532 cpu_manager.go:414] "containerMap: removing container" podUID="10b19b85-cfe2-4365-b9c6-b57dfb53ab24" containerName="istio-proxy" Oct 31 13:37:11.403033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.402877 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b19b85-cfe2-4365-b9c6-b57dfb53ab24" containerName="istio-proxy" Oct 31 13:37:11.403033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.402951 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ecad2629-f911-440f-ac75-74d7ef9cfdb6" containerName="istio-proxy" Oct 31 13:37:11.403033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.402959 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecad2629-f911-440f-ac75-74d7ef9cfdb6" containerName="istio-proxy" Oct 31 13:37:11.403033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.402997 5532 cpu_manager.go:414] "containerMap: removing container" podUID="48476992-f75d-4caf-aed3-02cd8c339644" containerName="istio-proxy" Oct 31 13:37:11.403033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.403005 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48476992-f75d-4caf-aed3-02cd8c339644" containerName="istio-proxy" Oct 31 13:37:11.403396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.403039 5532 cpu_manager.go:414] "containerMap: removing container" podUID="6e2aaa6e-c071-4d13-8aa9-857383962fdb" containerName="maas-api" Oct 31 13:37:11.403396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.403046 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e2aaa6e-c071-4d13-8aa9-857383962fdb" containerName="maas-api" Oct 31 13:37:11.403396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.403236 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecad2629-f911-440f-ac75-74d7ef9cfdb6" containerName="istio-proxy" Oct 31 13:37:11.403396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.403259 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="48476992-f75d-4caf-aed3-02cd8c339644" containerName="istio-proxy" Oct 31 13:37:11.403396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.403275 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e2aaa6e-c071-4d13-8aa9-857383962fdb" containerName="maas-api" Oct 31 13:37:11.403396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.403286 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b19b85-cfe2-4365-b9c6-b57dfb53ab24" containerName="istio-proxy" Oct 31 13:37:11.439336 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.439303 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv"] Oct 31 13:37:11.439513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.439432 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.536849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.536813 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0792431b-9ede-4c06-9d2e-29b8187e13ab-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.536849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.536863 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.537096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.536964 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.537096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.537000 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzw2\" (UniqueName: \"kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-kube-api-access-kbzw2\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.537202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.537083 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.537202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.537174 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.537314 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.537205 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.537504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.537485 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.537560 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.537521 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.639429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.639389 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.639429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.639426 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.639719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.639481 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0792431b-9ede-4c06-9d2e-29b8187e13ab-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.639719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.639501 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.639719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.639544 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.639719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.639571 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzw2\" (UniqueName: \"kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-kube-api-access-kbzw2\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.639719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.639622 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.639719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.639683 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.640054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.639721 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.640054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.639897 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.640054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.639956 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.640290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.640266 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.640359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.640317 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.640359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.640338 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0792431b-9ede-4c06-9d2e-29b8187e13ab-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.642163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.642135 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.642301 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.642228 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.653796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.653727 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.653912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.653895 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzw2\" (UniqueName: \"kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-kube-api-access-kbzw2\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:11.767530 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:11.765891 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:12.318966 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:37:12.317159 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0792431b_9ede_4c06_9d2e_29b8187e13ab.slice/crio-d0027832dfc345e6d01134d4953c4c6a250fc262e8ec4f18f417baed329932e5 WatchSource:0}: Error finding container d0027832dfc345e6d01134d4953c4c6a250fc262e8ec4f18f417baed329932e5: Status 404 returned error can't find the container with id d0027832dfc345e6d01134d4953c4c6a250fc262e8ec4f18f417baed329932e5 Oct 31 13:37:12.319238 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:12.319162 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv"] Oct 31 13:37:12.322364 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:12.322256 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:12.322364 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:12.322340 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:12.322643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:12.322379 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:12.504667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:12.504624 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" event={"ID":"0792431b-9ede-4c06-9d2e-29b8187e13ab","Type":"ContainerStarted","Data":"d1aac86f225dd13868c44eb0f5eae9d1fc42c532cde8c5a0915a96c393ed973c"} Oct 31 13:37:12.505127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:12.504696 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" event={"ID":"0792431b-9ede-4c06-9d2e-29b8187e13ab","Type":"ContainerStarted","Data":"d0027832dfc345e6d01134d4953c4c6a250fc262e8ec4f18f417baed329932e5"} Oct 31 13:37:12.545644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:12.545567 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" podStartSLOduration=1.5455454149999999 podStartE2EDuration="1.545545415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:12.53697823 +0000 UTC m=+2000.682722061" watchObservedRunningTime="2026-10-31 13:37:12.545545415 +0000 UTC m=+2000.691289247" Oct 31 13:37:12.802621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:12.789288 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:13.770531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:13.770483 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 13:37:13.770982 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:13.770558 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" podUID="0792431b-9ede-4c06-9d2e-29b8187e13ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 13:37:14.770034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:14.769994 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 13:37:14.770237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:14.770065 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" podUID="0792431b-9ede-4c06-9d2e-29b8187e13ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 13:37:15.768445 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:15.768408 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 13:37:15.768938 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:15.768465 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" podUID="0792431b-9ede-4c06-9d2e-29b8187e13ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 13:37:16.573394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:16.573343 5532 generic.go:356] "Generic (PLEG): container finished" podID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerID="0927e77e1dc6ff013793181212c82b0b1d6ec76025504df6f05368c223e76448" exitCode=0 Oct 31 13:37:16.574036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:16.573997 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" event={"ID":"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732","Type":"ContainerDied","Data":"0927e77e1dc6ff013793181212c82b0b1d6ec76025504df6f05368c223e76448"} Oct 31 13:37:16.574900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:16.574881 5532 scope.go:122] "RemoveContainer" containerID="0927e77e1dc6ff013793181212c82b0b1d6ec76025504df6f05368c223e76448" Oct 31 13:37:16.585620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:16.583791 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:16.585620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:16.583890 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:16.585620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:16.583935 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:16.780859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:16.780818 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 13:37:16.781571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:16.780893 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" podUID="0792431b-9ede-4c06-9d2e-29b8187e13ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 13:37:17.798961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:17.787442 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" event={"ID":"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732","Type":"ContainerStarted","Data":"7b13af30176895375fb1dc699eee8d87ec20f6cb3e2c55fce120c6bfc074846a"} Oct 31 13:37:17.798961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:17.782019 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 13:37:17.798961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:17.791636 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" podUID="0792431b-9ede-4c06-9d2e-29b8187e13ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 13:37:18.084094 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:18.084001 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:37:18.084453 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:18.084361 5532 patch_prober.go:29] interesting pod/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:37:18.084627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:18.084578 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 13:37:18.837484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:18.837448 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 13:37:18.838115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:18.838080 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" podUID="0792431b-9ede-4c06-9d2e-29b8187e13ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 13:37:19.088679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:19.085973 5532 patch_prober.go:29] interesting pod/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:37:19.088679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:19.086034 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 13:37:19.771618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:19.770620 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Oct 31 13:37:19.771618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:19.770697 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" podUID="0792431b-9ede-4c06-9d2e-29b8187e13ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Oct 31 13:37:20.096643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:20.095225 5532 patch_prober.go:29] interesting pod/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:37:20.096643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:20.095297 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 13:37:20.779385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:20.779352 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:20.779859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:20.779844 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:20.780624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:20.780587 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:37:21.084115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:21.084027 5532 patch_prober.go:29] interesting pod/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:37:21.084115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:21.084077 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 13:37:22.084232 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:22.084200 5532 patch_prober.go:29] interesting pod/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:37:22.084695 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:22.084267 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 13:37:22.885965 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:22.885922 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:37:23.084304 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:23.084266 5532 patch_prober.go:29] interesting pod/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:37:23.084742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:23.084324 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 13:37:24.083836 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:24.083800 5532 patch_prober.go:29] interesting pod/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" start-of-body= Oct 31 13:37:24.084016 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:24.083860 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.151:15021/healthz/ready\": dial tcp 10.132.0.151:15021: connect: connection refused" Oct 31 13:37:25.089936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:25.089904 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:37:25.092333 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:25.092307 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:37:26.463606 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.463568 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg"] Oct 31 13:37:26.468139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.468119 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.492733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.492582 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg"] Oct 31 13:37:26.590663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.590552 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.590663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.590590 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.590663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.590627 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.590982 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.590863 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.590982 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.590963 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.591093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.591028 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.591294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.591274 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7zns\" (UniqueName: \"kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-kube-api-access-h7zns\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.591392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.591351 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.591457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.591417 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.693038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.693000 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.693038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.693041 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.693311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.693058 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.693311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.693181 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.693311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.693208 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.693311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.693244 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.693539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.693357 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7zns\" (UniqueName: \"kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-kube-api-access-h7zns\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.693539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.693414 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.693539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.693432 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.693539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.693488 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.693771 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.693546 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-data\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.693856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.693833 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-certs\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.694146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.694120 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-credential-socket\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.694146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.694138 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istiod-ca-cert\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.695572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.695550 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-envoy\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.696117 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.696094 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-podinfo\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.707065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.707043 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-token\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.707541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.707519 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7zns\" (UniqueName: \"kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-kube-api-access-h7zns\") pod \"e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:26.783179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:26.783095 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:27.080707 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:27.080676 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg"] Oct 31 13:37:27.081187 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:37:27.081160 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d4a1f6_d0cd_4ce9_8ea6_68c6550da758.slice/crio-7f2c8aa18e36d51137818b9806e373c4d05bf26399df9598f14886e41f5b9ff9 WatchSource:0}: Error finding container 7f2c8aa18e36d51137818b9806e373c4d05bf26399df9598f14886e41f5b9ff9: Status 404 returned error can't find the container with id 7f2c8aa18e36d51137818b9806e373c4d05bf26399df9598f14886e41f5b9ff9 Oct 31 13:37:27.084579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:27.084548 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:27.084706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:27.084657 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:27.084706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:27.084699 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:37:27.469632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:27.467368 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp"] Oct 31 13:37:27.469632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:27.467631 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" podUID="9040d659-6f09-4fc7-a4b3-193870beb6c2" containerName="maas-api" containerID="cri-o://364ac701927cba1693f5f452bfb15fd8685fce01c0dfaeb360fc6c0dd1807dd6" gracePeriod=30 Oct 31 13:37:28.232182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.230862 5532 generic.go:356] "Generic (PLEG): container finished" podID="9040d659-6f09-4fc7-a4b3-193870beb6c2" containerID="364ac701927cba1693f5f452bfb15fd8685fce01c0dfaeb360fc6c0dd1807dd6" exitCode=0 Oct 31 13:37:28.232182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.231082 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" event={"ID":"9040d659-6f09-4fc7-a4b3-193870beb6c2","Type":"ContainerDied","Data":"364ac701927cba1693f5f452bfb15fd8685fce01c0dfaeb360fc6c0dd1807dd6"} Oct 31 13:37:28.248190 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.246417 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" event={"ID":"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758","Type":"ContainerStarted","Data":"befdafdde9821cbd61f761ea6758e7a17197ff65cb4ef37dcd91e34f294e9875"} Oct 31 13:37:28.248190 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.246481 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" event={"ID":"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758","Type":"ContainerStarted","Data":"7f2c8aa18e36d51137818b9806e373c4d05bf26399df9598f14886e41f5b9ff9"} Oct 31 13:37:28.270628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.270575 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:37:28.284569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.284403 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" podStartSLOduration=2.284382813 podStartE2EDuration="2.284382813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:28.275465823 +0000 UTC m=+2016.421209651" watchObservedRunningTime="2026-10-31 13:37:28.284382813 +0000 UTC m=+2016.430126645" Oct 31 13:37:28.433654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.433587 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-metrics-certs\") pod \"9040d659-6f09-4fc7-a4b3-193870beb6c2\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " Oct 31 13:37:28.433877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.433738 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-maas-api-tls\") pod \"9040d659-6f09-4fc7-a4b3-193870beb6c2\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " Oct 31 13:37:28.433877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.433851 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9040d659-6f09-4fc7-a4b3-193870beb6c2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9040d659-6f09-4fc7-a4b3-193870beb6c2-openshift-service-ca\") pod \"9040d659-6f09-4fc7-a4b3-193870beb6c2\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " Oct 31 13:37:28.434223 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.434186 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9040d659-6f09-4fc7-a4b3-193870beb6c2-kube-api-access-78xlc\" (UniqueName: \"kubernetes.io/projected/9040d659-6f09-4fc7-a4b3-193870beb6c2-kube-api-access-78xlc\") pod \"9040d659-6f09-4fc7-a4b3-193870beb6c2\" (UID: \"9040d659-6f09-4fc7-a4b3-193870beb6c2\") " Oct 31 13:37:28.435320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.435292 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9040d659-6f09-4fc7-a4b3-193870beb6c2-openshift-service-ca" pod "9040d659-6f09-4fc7-a4b3-193870beb6c2" (UID: "9040d659-6f09-4fc7-a4b3-193870beb6c2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:28.447666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.438502 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-metrics-certs" pod "9040d659-6f09-4fc7-a4b3-193870beb6c2" (UID: "9040d659-6f09-4fc7-a4b3-193870beb6c2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:28.473943 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.470255 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-maas-api-tls" pod "9040d659-6f09-4fc7-a4b3-193870beb6c2" (UID: "9040d659-6f09-4fc7-a4b3-193870beb6c2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:28.492494 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.485714 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9040d659-6f09-4fc7-a4b3-193870beb6c2-kube-api-access-78xlc" pod "9040d659-6f09-4fc7-a4b3-193870beb6c2" (UID: "9040d659-6f09-4fc7-a4b3-193870beb6c2"). InnerVolumeSpecName "kube-api-access-78xlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:28.613618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.601886 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78xlc\" (UniqueName: \"kubernetes.io/projected/9040d659-6f09-4fc7-a4b3-193870beb6c2-kube-api-access-78xlc\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:28.613618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.601921 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:28.613618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.602139 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9040d659-6f09-4fc7-a4b3-193870beb6c2-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:28.613618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.602158 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9040d659-6f09-4fc7-a4b3-193870beb6c2-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:28.790166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.789685 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:28.790166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.789815 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 13:37:28.790166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:28.789855 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" podUID="68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 13:37:29.292803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:29.292770 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" Oct 31 13:37:29.292963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:29.292804 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp" event={"ID":"9040d659-6f09-4fc7-a4b3-193870beb6c2","Type":"ContainerDied","Data":"ed32aff5806a63adff5d2f219c4141311180f9046743d905d09da52a91097530"} Oct 31 13:37:29.292963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:29.292877 5532 scope.go:122] "RemoveContainer" containerID="364ac701927cba1693f5f452bfb15fd8685fce01c0dfaeb360fc6c0dd1807dd6" Oct 31 13:37:29.324524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:29.324497 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp"] Oct 31 13:37:29.331008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:29.330985 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-580f24-dc944ddfb-fjssp"] Oct 31 13:37:29.784404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:29.784369 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 13:37:29.784825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:29.784430 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" podUID="68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 13:37:30.016869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:30.016836 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9040d659-6f09-4fc7-a4b3-193870beb6c2" path="/var/lib/kubelet/pods/9040d659-6f09-4fc7-a4b3-193870beb6c2/volumes" Oct 31 13:37:30.784067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:30.784024 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 13:37:30.784254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:30.784085 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" podUID="68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 13:37:31.785522 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:31.783749 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 13:37:31.785522 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:31.783802 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" podUID="68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 13:37:31.789969 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:37:31.789924 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:37:32.798617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:32.791989 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 13:37:32.798617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:32.792148 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" podUID="68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 13:37:33.783987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:33.783953 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" start-of-body= Oct 31 13:37:33.784318 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:33.784294 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" podUID="68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.160:15021/healthz/ready\": dial tcp 10.132.0.160:15021: connect: connection refused" Oct 31 13:37:34.812809 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:34.812771 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:34.815444 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:34.815410 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:34.816561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:34.816532 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:37:48.592453 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:48.592390 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d"] Oct 31 13:37:49.606321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:49.606237 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:37:52.158735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.158696 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc"] Oct 31 13:37:52.159948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.159918 5532 cpu_manager.go:414] "containerMap: removing container" podUID="9040d659-6f09-4fc7-a4b3-193870beb6c2" containerName="maas-api" Oct 31 13:37:52.159948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.159946 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9040d659-6f09-4fc7-a4b3-193870beb6c2" containerName="maas-api" Oct 31 13:37:52.160240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.160226 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="9040d659-6f09-4fc7-a4b3-193870beb6c2" containerName="maas-api" Oct 31 13:37:52.165260 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.165238 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" Oct 31 13:37:52.175637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.174725 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc"] Oct 31 13:37:52.175637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.174984 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:37:52.176451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.176428 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:37:52.319787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.319748 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc\" (UID: \"f1fc28fd-67d3-4e77-8555-9e51fb406358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" Oct 31 13:37:52.320016 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.319993 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v95b5\" (UniqueName: \"kubernetes.io/projected/f1fc28fd-67d3-4e77-8555-9e51fb406358-kube-api-access-v95b5\") pod \"maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc\" (UID: \"f1fc28fd-67d3-4e77-8555-9e51fb406358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" Oct 31 13:37:52.320144 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.320123 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc\" (UID: \"f1fc28fd-67d3-4e77-8555-9e51fb406358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" Oct 31 13:37:52.422396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.422293 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc\" (UID: \"f1fc28fd-67d3-4e77-8555-9e51fb406358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" Oct 31 13:37:52.422582 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.422531 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v95b5\" (UniqueName: \"kubernetes.io/projected/f1fc28fd-67d3-4e77-8555-9e51fb406358-kube-api-access-v95b5\") pod \"maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc\" (UID: \"f1fc28fd-67d3-4e77-8555-9e51fb406358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" Oct 31 13:37:52.422720 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.422641 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc\" (UID: \"f1fc28fd-67d3-4e77-8555-9e51fb406358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" Oct 31 13:37:52.423446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.423422 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc\" (UID: \"f1fc28fd-67d3-4e77-8555-9e51fb406358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" Oct 31 13:37:52.425749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.425726 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc\" (UID: \"f1fc28fd-67d3-4e77-8555-9e51fb406358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" Oct 31 13:37:52.436189 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.436164 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v95b5\" (UniqueName: \"kubernetes.io/projected/f1fc28fd-67d3-4e77-8555-9e51fb406358-kube-api-access-v95b5\") pod \"maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc\" (UID: \"f1fc28fd-67d3-4e77-8555-9e51fb406358\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" Oct 31 13:37:52.524544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.524512 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" Oct 31 13:37:52.816028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:52.815996 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc"] Oct 31 13:37:52.816583 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:37:52.816558 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fc28fd_67d3_4e77_8555_9e51fb406358.slice/crio-883ecca6fc758d90eb49c670464692f0999553b4bf95398819bc3da04dc8e2aa WatchSource:0}: Error finding container 883ecca6fc758d90eb49c670464692f0999553b4bf95398819bc3da04dc8e2aa: Status 404 returned error can't find the container with id 883ecca6fc758d90eb49c670464692f0999553b4bf95398819bc3da04dc8e2aa Oct 31 13:37:53.593420 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:53.593361 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" podUID="ef722ded-585d-4913-9691-aa5170f51a15" containerName="payload-pre-processing" containerID="cri-o://5c1a0a3e6ea673efc6bc67b0cac153a0c8ab0e9b530d546d337fb9cbcaa69b1c" gracePeriod=25 Oct 31 13:37:53.612405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:53.612375 5532 scope.go:122] "RemoveContainer" containerID="12c577584e720b4b6a25e716b5c64ebb401d4d73e7e1a10fa40626e69e878379" Oct 31 13:37:53.707098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:53.707074 5532 scope.go:122] "RemoveContainer" containerID="5d8a4afda6275c49a05d4651ad61e4f32bb13ca7362ee4b70d3312a58df2fe82" Oct 31 13:37:53.833546 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:53.833507 5532 generic.go:356] "Generic (PLEG): container finished" podID="ef722ded-585d-4913-9691-aa5170f51a15" containerID="5c1a0a3e6ea673efc6bc67b0cac153a0c8ab0e9b530d546d337fb9cbcaa69b1c" exitCode=0 Oct 31 13:37:53.833779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:53.833563 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" event={"ID":"ef722ded-585d-4913-9691-aa5170f51a15","Type":"ContainerDied","Data":"5c1a0a3e6ea673efc6bc67b0cac153a0c8ab0e9b530d546d337fb9cbcaa69b1c"} Oct 31 13:37:53.835728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:53.835697 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" event={"ID":"f1fc28fd-67d3-4e77-8555-9e51fb406358","Type":"ContainerStarted","Data":"1a12ca69ccc8db0bbb70491c3f93d13cc9d8a8d0103c477d56687a8b2cd8509c"} Oct 31 13:37:53.835862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:53.835762 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" event={"ID":"f1fc28fd-67d3-4e77-8555-9e51fb406358","Type":"ContainerStarted","Data":"883ecca6fc758d90eb49c670464692f0999553b4bf95398819bc3da04dc8e2aa"} Oct 31 13:37:53.860685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:53.860634 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" podStartSLOduration=1.860617449 podStartE2EDuration="1.860617449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:37:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:37:53.857729104 +0000 UTC m=+2042.003472932" watchObservedRunningTime="2026-10-31 13:37:53.860617449 +0000 UTC m=+2042.006361278" Oct 31 13:37:53.937328 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:53.937297 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" Oct 31 13:37:54.050816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:54.050784 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef722ded-585d-4913-9691-aa5170f51a15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef722ded-585d-4913-9691-aa5170f51a15-plugins-config-volume\") pod \"ef722ded-585d-4913-9691-aa5170f51a15\" (UID: \"ef722ded-585d-4913-9691-aa5170f51a15\") " Oct 31 13:37:54.050979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:54.050881 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef722ded-585d-4913-9691-aa5170f51a15-kube-api-access-ppd9j\" (UniqueName: \"kubernetes.io/projected/ef722ded-585d-4913-9691-aa5170f51a15-kube-api-access-ppd9j\") pod \"ef722ded-585d-4913-9691-aa5170f51a15\" (UID: \"ef722ded-585d-4913-9691-aa5170f51a15\") " Oct 31 13:37:54.051150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:54.051128 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef722ded-585d-4913-9691-aa5170f51a15-plugins-config-volume" pod "ef722ded-585d-4913-9691-aa5170f51a15" (UID: "ef722ded-585d-4913-9691-aa5170f51a15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:54.051539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:54.051524 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ef722ded-585d-4913-9691-aa5170f51a15-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:54.053095 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:54.053074 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef722ded-585d-4913-9691-aa5170f51a15-kube-api-access-ppd9j" pod "ef722ded-585d-4913-9691-aa5170f51a15" (UID: "ef722ded-585d-4913-9691-aa5170f51a15"). InnerVolumeSpecName "kube-api-access-ppd9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:54.153645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:54.153567 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppd9j\" (UniqueName: \"kubernetes.io/projected/ef722ded-585d-4913-9691-aa5170f51a15-kube-api-access-ppd9j\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:54.845154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:54.845119 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" Oct 31 13:37:54.845613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:54.845152 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d" event={"ID":"ef722ded-585d-4913-9691-aa5170f51a15","Type":"ContainerDied","Data":"6ff9c0ce82a6a1ae0626510f14ece48be761de069fde04e9c96f8050ba43a78a"} Oct 31 13:37:54.845613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:54.845210 5532 scope.go:122] "RemoveContainer" containerID="5c1a0a3e6ea673efc6bc67b0cac153a0c8ab0e9b530d546d337fb9cbcaa69b1c" Oct 31 13:37:54.846979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:54.846957 5532 generic.go:356] "Generic (PLEG): container finished" podID="f1fc28fd-67d3-4e77-8555-9e51fb406358" containerID="1a12ca69ccc8db0bbb70491c3f93d13cc9d8a8d0103c477d56687a8b2cd8509c" exitCode=0 Oct 31 13:37:54.847070 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:54.846987 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" event={"ID":"f1fc28fd-67d3-4e77-8555-9e51fb406358","Type":"ContainerDied","Data":"1a12ca69ccc8db0bbb70491c3f93d13cc9d8a8d0103c477d56687a8b2cd8509c"} Oct 31 13:37:54.888543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:54.888470 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d"] Oct 31 13:37:54.892955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:54.892931 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-cbb3d5-77cb59c59-m7w8d"] Oct 31 13:37:56.005195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.005164 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef722ded-585d-4913-9691-aa5170f51a15" path="/var/lib/kubelet/pods/ef722ded-585d-4913-9691-aa5170f51a15/volumes" Oct 31 13:37:56.031552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.031529 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" Oct 31 13:37:56.089585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.089544 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-xks-ca\") pod \"f1fc28fd-67d3-4e77-8555-9e51fb406358\" (UID: \"f1fc28fd-67d3-4e77-8555-9e51fb406358\") " Oct 31 13:37:56.089786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.089761 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-openshift-ca\") pod \"f1fc28fd-67d3-4e77-8555-9e51fb406358\" (UID: \"f1fc28fd-67d3-4e77-8555-9e51fb406358\") " Oct 31 13:37:56.089922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.089889 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1fc28fd-67d3-4e77-8555-9e51fb406358-kube-api-access-v95b5\" (UniqueName: \"kubernetes.io/projected/f1fc28fd-67d3-4e77-8555-9e51fb406358-kube-api-access-v95b5\") pod \"f1fc28fd-67d3-4e77-8555-9e51fb406358\" (UID: \"f1fc28fd-67d3-4e77-8555-9e51fb406358\") " Oct 31 13:37:56.090127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.090106 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-openshift-ca" pod "f1fc28fd-67d3-4e77-8555-9e51fb406358" (UID: "f1fc28fd-67d3-4e77-8555-9e51fb406358"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:37:56.090921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.090828 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:56.092645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.091963 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-xks-ca" pod "f1fc28fd-67d3-4e77-8555-9e51fb406358" (UID: "f1fc28fd-67d3-4e77-8555-9e51fb406358"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:37:56.092645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.092312 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1fc28fd-67d3-4e77-8555-9e51fb406358-kube-api-access-v95b5" pod "f1fc28fd-67d3-4e77-8555-9e51fb406358" (UID: "f1fc28fd-67d3-4e77-8555-9e51fb406358"). InnerVolumeSpecName "kube-api-access-v95b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:37:56.193029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.192985 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v95b5\" (UniqueName: \"kubernetes.io/projected/f1fc28fd-67d3-4e77-8555-9e51fb406358-kube-api-access-v95b5\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:56.193029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.193016 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1fc28fd-67d3-4e77-8555-9e51fb406358-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:37:56.865045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.865015 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" Oct 31 13:37:56.865045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.865042 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc" event={"ID":"f1fc28fd-67d3-4e77-8555-9e51fb406358","Type":"ContainerDied","Data":"883ecca6fc758d90eb49c670464692f0999553b4bf95398819bc3da04dc8e2aa"} Oct 31 13:37:56.865272 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:37:56.865085 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="883ecca6fc758d90eb49c670464692f0999553b4bf95398819bc3da04dc8e2aa" Oct 31 13:38:01.763732 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:38:01.763687 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:38:02.173055 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:02.173023 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb"] Oct 31 13:38:06.188667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.188625 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68"] Oct 31 13:38:06.189706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.189683 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ef722ded-585d-4913-9691-aa5170f51a15" containerName="payload-pre-processing" Oct 31 13:38:06.189706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.189706 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef722ded-585d-4913-9691-aa5170f51a15" containerName="payload-pre-processing" Oct 31 13:38:06.189978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.189751 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f1fc28fd-67d3-4e77-8555-9e51fb406358" containerName="revoke-keys" Oct 31 13:38:06.189978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.189760 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1fc28fd-67d3-4e77-8555-9e51fb406358" containerName="revoke-keys" Oct 31 13:38:06.189978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.189950 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef722ded-585d-4913-9691-aa5170f51a15" containerName="payload-pre-processing" Oct 31 13:38:06.189978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.189975 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1fc28fd-67d3-4e77-8555-9e51fb406358" containerName="revoke-keys" Oct 31 13:38:06.201361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.194677 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.208653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.208627 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-61da9d-serving-cert\"" Oct 31 13:38:06.216506 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.216479 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68"] Oct 31 13:38:06.326604 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.326559 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38485377-d08e-459c-a544-50cf06cfe529-openshift-service-ca\") pod \"maas-api-e2e-inf-a-61da9d-77fb898559-tlv68\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.326788 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.326777 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-maas-api-tls\") pod \"maas-api-e2e-inf-a-61da9d-77fb898559-tlv68\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.326902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.326886 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqd58\" (UniqueName: \"kubernetes.io/projected/38485377-d08e-459c-a544-50cf06cfe529-kube-api-access-cqd58\") pod \"maas-api-e2e-inf-a-61da9d-77fb898559-tlv68\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.327008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.326958 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-metrics-certs\") pod \"maas-api-e2e-inf-a-61da9d-77fb898559-tlv68\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.428563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.428526 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38485377-d08e-459c-a544-50cf06cfe529-openshift-service-ca\") pod \"maas-api-e2e-inf-a-61da9d-77fb898559-tlv68\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.428819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.428743 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-maas-api-tls\") pod \"maas-api-e2e-inf-a-61da9d-77fb898559-tlv68\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.428888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.428849 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqd58\" (UniqueName: \"kubernetes.io/projected/38485377-d08e-459c-a544-50cf06cfe529-kube-api-access-cqd58\") pod \"maas-api-e2e-inf-a-61da9d-77fb898559-tlv68\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.428944 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.428920 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-metrics-certs\") pod \"maas-api-e2e-inf-a-61da9d-77fb898559-tlv68\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.429453 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.429424 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38485377-d08e-459c-a544-50cf06cfe529-openshift-service-ca\") pod \"maas-api-e2e-inf-a-61da9d-77fb898559-tlv68\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.431559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.431534 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-maas-api-tls\") pod \"maas-api-e2e-inf-a-61da9d-77fb898559-tlv68\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.431943 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.431920 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-metrics-certs\") pod \"maas-api-e2e-inf-a-61da9d-77fb898559-tlv68\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.443844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.443779 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqd58\" (UniqueName: \"kubernetes.io/projected/38485377-d08e-459c-a544-50cf06cfe529-kube-api-access-cqd58\") pod \"maas-api-e2e-inf-a-61da9d-77fb898559-tlv68\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.529763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.529725 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:06.829980 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:38:06.829942 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38485377_d08e_459c_a544_50cf06cfe529.slice/crio-5cbfc6fc5ff06385a2a9de93679f7f8d9f9e5fed2783cea7c3151fd39a677d58 WatchSource:0}: Error finding container 5cbfc6fc5ff06385a2a9de93679f7f8d9f9e5fed2783cea7c3151fd39a677d58: Status 404 returned error can't find the container with id 5cbfc6fc5ff06385a2a9de93679f7f8d9f9e5fed2783cea7c3151fd39a677d58 Oct 31 13:38:06.838463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.832833 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68"] Oct 31 13:38:06.838463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.834330 5532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:38:06.975615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:06.972299 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" event={"ID":"38485377-d08e-459c-a544-50cf06cfe529","Type":"ContainerStarted","Data":"5cbfc6fc5ff06385a2a9de93679f7f8d9f9e5fed2783cea7c3151fd39a677d58"} Oct 31 13:38:07.174104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.174060 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" podUID="cec34c9a-e736-4f27-a15d-14532e0e5d6c" containerName="payload-processing" containerID="cri-o://c99e3cd4f9498c6df33f3a5cb99bc717739d9a5bcc5999bf840b90323c5b93a0" gracePeriod=25 Oct 31 13:38:07.544512 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.544478 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" Oct 31 13:38:07.572484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.572445 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec34c9a-e736-4f27-a15d-14532e0e5d6c-kube-api-access-brg2w\" (UniqueName: \"kubernetes.io/projected/cec34c9a-e736-4f27-a15d-14532e0e5d6c-kube-api-access-brg2w\") pod \"cec34c9a-e736-4f27-a15d-14532e0e5d6c\" (UID: \"cec34c9a-e736-4f27-a15d-14532e0e5d6c\") " Oct 31 13:38:07.573488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.572811 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec34c9a-e736-4f27-a15d-14532e0e5d6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec34c9a-e736-4f27-a15d-14532e0e5d6c-plugins-config-volume\") pod \"cec34c9a-e736-4f27-a15d-14532e0e5d6c\" (UID: \"cec34c9a-e736-4f27-a15d-14532e0e5d6c\") " Oct 31 13:38:07.574371 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.574145 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec34c9a-e736-4f27-a15d-14532e0e5d6c-plugins-config-volume" pod "cec34c9a-e736-4f27-a15d-14532e0e5d6c" (UID: "cec34c9a-e736-4f27-a15d-14532e0e5d6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:07.579275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.579244 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec34c9a-e736-4f27-a15d-14532e0e5d6c-kube-api-access-brg2w" pod "cec34c9a-e736-4f27-a15d-14532e0e5d6c" (UID: "cec34c9a-e736-4f27-a15d-14532e0e5d6c"). InnerVolumeSpecName "kube-api-access-brg2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:07.676416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.676383 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cec34c9a-e736-4f27-a15d-14532e0e5d6c-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:07.676416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.676412 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brg2w\" (UniqueName: \"kubernetes.io/projected/cec34c9a-e736-4f27-a15d-14532e0e5d6c-kube-api-access-brg2w\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:07.981818 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.981781 5532 generic.go:356] "Generic (PLEG): container finished" podID="cec34c9a-e736-4f27-a15d-14532e0e5d6c" containerID="c99e3cd4f9498c6df33f3a5cb99bc717739d9a5bcc5999bf840b90323c5b93a0" exitCode=0 Oct 31 13:38:07.982006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.981829 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" Oct 31 13:38:07.982006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.981831 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" event={"ID":"cec34c9a-e736-4f27-a15d-14532e0e5d6c","Type":"ContainerDied","Data":"c99e3cd4f9498c6df33f3a5cb99bc717739d9a5bcc5999bf840b90323c5b93a0"} Oct 31 13:38:07.982006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.981890 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb" event={"ID":"cec34c9a-e736-4f27-a15d-14532e0e5d6c","Type":"ContainerDied","Data":"cb1c6d83b02fd03628ab4840ef0b58c681a8875a905f7ccdea86f4a2cb74c14f"} Oct 31 13:38:07.982006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.981913 5532 scope.go:122] "RemoveContainer" containerID="c99e3cd4f9498c6df33f3a5cb99bc717739d9a5bcc5999bf840b90323c5b93a0" Oct 31 13:38:07.984102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.984034 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" event={"ID":"38485377-d08e-459c-a544-50cf06cfe529","Type":"ContainerStarted","Data":"1fe824733ecd7701b3fbd21a000b4c540d190cd61d6dd8705d4d2586214c0dd5"} Oct 31 13:38:07.984102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.984101 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:07.993935 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.993919 5532 scope.go:122] "RemoveContainer" containerID="c99e3cd4f9498c6df33f3a5cb99bc717739d9a5bcc5999bf840b90323c5b93a0" Oct 31 13:38:07.994185 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:38:07.994165 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c99e3cd4f9498c6df33f3a5cb99bc717739d9a5bcc5999bf840b90323c5b93a0\": container with ID starting with c99e3cd4f9498c6df33f3a5cb99bc717739d9a5bcc5999bf840b90323c5b93a0 not found: ID does not exist" containerID="c99e3cd4f9498c6df33f3a5cb99bc717739d9a5bcc5999bf840b90323c5b93a0" Oct 31 13:38:07.994272 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:07.994192 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c99e3cd4f9498c6df33f3a5cb99bc717739d9a5bcc5999bf840b90323c5b93a0"} err="failed to get container status \"c99e3cd4f9498c6df33f3a5cb99bc717739d9a5bcc5999bf840b90323c5b93a0\": rpc error: code = NotFound desc = could not find container \"c99e3cd4f9498c6df33f3a5cb99bc717739d9a5bcc5999bf840b90323c5b93a0\": container with ID starting with c99e3cd4f9498c6df33f3a5cb99bc717739d9a5bcc5999bf840b90323c5b93a0 not found: ID does not exist" Oct 31 13:38:08.011382 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:08.011341 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" podStartSLOduration=1.5815467989999998 podStartE2EDuration="2.011327825s" totalImagesPullingTime="429.781026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:08.008437011 +0000 UTC m=+2056.154180840" watchObservedRunningTime="2026-10-31 13:38:08.011327825 +0000 UTC m=+2056.157071678" Oct 31 13:38:08.030707 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:08.030680 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb"] Oct 31 13:38:08.034919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:08.034888 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-c45a37-58bcf5bf75-846fb"] Oct 31 13:38:10.008172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:10.008134 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec34c9a-e736-4f27-a15d-14532e0e5d6c" path="/var/lib/kubelet/pods/cec34c9a-e736-4f27-a15d-14532e0e5d6c/volumes" Oct 31 13:38:14.006554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:14.006527 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:38:15.391902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.391870 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg"] Oct 31 13:38:15.392733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.392717 5532 cpu_manager.go:414] "containerMap: removing container" podUID="cec34c9a-e736-4f27-a15d-14532e0e5d6c" containerName="payload-processing" Oct 31 13:38:15.392790 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.392733 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec34c9a-e736-4f27-a15d-14532e0e5d6c" containerName="payload-processing" Oct 31 13:38:15.392869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.392859 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec34c9a-e736-4f27-a15d-14532e0e5d6c" containerName="payload-processing" Oct 31 13:38:15.406224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.406181 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" Oct 31 13:38:15.416337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.416305 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-cae9ec\"" Oct 31 13:38:15.420778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.420753 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg"] Oct 31 13:38:15.427441 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.427412 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw"] Oct 31 13:38:15.429971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.429945 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-cae9ec-dockercfg-kg487\"" Oct 31 13:38:15.434759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.434737 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" Oct 31 13:38:15.451219 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.451188 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw"] Oct 31 13:38:15.503185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.503141 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t89pq\" (UniqueName: \"kubernetes.io/projected/79003251-6684-4494-9076-fb98c4ac43de-kube-api-access-t89pq\") pod \"payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw\" (UID: \"79003251-6684-4494-9076-fb98c4ac43de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" Oct 31 13:38:15.503375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.503200 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gjxv\" (UniqueName: \"kubernetes.io/projected/f462da06-0178-4e54-b9bd-ad83be10daf3-kube-api-access-2gjxv\") pod \"payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg\" (UID: \"f462da06-0178-4e54-b9bd-ad83be10daf3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" Oct 31 13:38:15.503502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.503478 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f462da06-0178-4e54-b9bd-ad83be10daf3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg\" (UID: \"f462da06-0178-4e54-b9bd-ad83be10daf3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" Oct 31 13:38:15.503565 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.503526 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79003251-6684-4494-9076-fb98c4ac43de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw\" (UID: \"79003251-6684-4494-9076-fb98c4ac43de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" Oct 31 13:38:15.611417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.611374 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f462da06-0178-4e54-b9bd-ad83be10daf3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg\" (UID: \"f462da06-0178-4e54-b9bd-ad83be10daf3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" Oct 31 13:38:15.613217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.613165 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79003251-6684-4494-9076-fb98c4ac43de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw\" (UID: \"79003251-6684-4494-9076-fb98c4ac43de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" Oct 31 13:38:15.613579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.613558 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t89pq\" (UniqueName: \"kubernetes.io/projected/79003251-6684-4494-9076-fb98c4ac43de-kube-api-access-t89pq\") pod \"payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw\" (UID: \"79003251-6684-4494-9076-fb98c4ac43de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" Oct 31 13:38:15.613775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.613757 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gjxv\" (UniqueName: \"kubernetes.io/projected/f462da06-0178-4e54-b9bd-ad83be10daf3-kube-api-access-2gjxv\") pod \"payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg\" (UID: \"f462da06-0178-4e54-b9bd-ad83be10daf3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" Oct 31 13:38:15.614519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.613086 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f462da06-0178-4e54-b9bd-ad83be10daf3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg\" (UID: \"f462da06-0178-4e54-b9bd-ad83be10daf3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" Oct 31 13:38:15.615287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.615261 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79003251-6684-4494-9076-fb98c4ac43de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw\" (UID: \"79003251-6684-4494-9076-fb98c4ac43de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" Oct 31 13:38:15.637613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.637304 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gjxv\" (UniqueName: \"kubernetes.io/projected/f462da06-0178-4e54-b9bd-ad83be10daf3-kube-api-access-2gjxv\") pod \"payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg\" (UID: \"f462da06-0178-4e54-b9bd-ad83be10daf3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" Oct 31 13:38:15.646409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.646380 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t89pq\" (UniqueName: \"kubernetes.io/projected/79003251-6684-4494-9076-fb98c4ac43de-kube-api-access-t89pq\") pod \"payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw\" (UID: \"79003251-6684-4494-9076-fb98c4ac43de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" Oct 31 13:38:15.738014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.737980 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" Oct 31 13:38:15.763090 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:15.763061 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" Oct 31 13:38:16.242693 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:16.236659 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg"] Oct 31 13:38:16.445283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:16.445228 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw"] Oct 31 13:38:17.082942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:17.082903 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" event={"ID":"79003251-6684-4494-9076-fb98c4ac43de","Type":"ContainerStarted","Data":"18bdf0a0282a04f0cc9e45d974cbcfc1ce46738955840da92a01d7ca96cab01c"} Oct 31 13:38:17.083150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:17.082971 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" event={"ID":"79003251-6684-4494-9076-fb98c4ac43de","Type":"ContainerStarted","Data":"a8a0242b2c35dc1b66300971586d641bf7dd6908920bb716e9aac17edc289dc3"} Oct 31 13:38:17.083150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:17.083004 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" Oct 31 13:38:17.084793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:17.084766 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" event={"ID":"f462da06-0178-4e54-b9bd-ad83be10daf3","Type":"ContainerStarted","Data":"d87006ae1a01ce00767bee32dbfc8b1baef3314e3f85b72edbd9f4a823a37e9b"} Oct 31 13:38:17.084921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:17.084812 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" event={"ID":"f462da06-0178-4e54-b9bd-ad83be10daf3","Type":"ContainerStarted","Data":"691019ec66ff5204af56242cc5417103860b13f93d6135704c87b7633a42eee6"} Oct 31 13:38:17.084921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:17.084835 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" Oct 31 13:38:17.108492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:17.108260 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" podStartSLOduration=2.10824524 podStartE2EDuration="2.10824524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:17.107280623 +0000 UTC m=+2065.253024450" watchObservedRunningTime="2026-10-31 13:38:17.10824524 +0000 UTC m=+2065.253989068" Oct 31 13:38:17.129694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:17.129641 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" podStartSLOduration=2.129625302 podStartE2EDuration="2.129625302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:17.128301726 +0000 UTC m=+2065.274045554" watchObservedRunningTime="2026-10-31 13:38:17.129625302 +0000 UTC m=+2065.275369131" Oct 31 13:38:21.939853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:21.939819 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs"] Oct 31 13:38:21.944770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:21.944752 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" Oct 31 13:38:21.958661 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:21.958635 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-61f295\"" Oct 31 13:38:21.961244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:21.961222 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs"] Oct 31 13:38:21.988443 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:21.988410 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q"] Oct 31 13:38:21.994646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:21.994607 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" Oct 31 13:38:22.025016 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.000005 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-61f295-dockercfg-nmd7x\"" Oct 31 13:38:22.040209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.040120 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q"] Oct 31 13:38:22.210451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.205053 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4f4e60-7bc8-4d83-a6eb-97812b694793-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs\" (UID: \"df4f4e60-7bc8-4d83-a6eb-97812b694793\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" Oct 31 13:38:22.210451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.205235 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245qz\" (UniqueName: \"kubernetes.io/projected/df4f4e60-7bc8-4d83-a6eb-97812b694793-kube-api-access-245qz\") pod \"payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs\" (UID: \"df4f4e60-7bc8-4d83-a6eb-97812b694793\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" Oct 31 13:38:22.210451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.205321 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q\" (UID: \"402e298e-b8a6-4cef-8fa4-a5e667a33ca4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" Oct 31 13:38:22.210451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.205376 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgtj7\" (UniqueName: \"kubernetes.io/projected/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-kube-api-access-mgtj7\") pod \"payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q\" (UID: \"402e298e-b8a6-4cef-8fa4-a5e667a33ca4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" Oct 31 13:38:22.315091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.312613 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4f4e60-7bc8-4d83-a6eb-97812b694793-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs\" (UID: \"df4f4e60-7bc8-4d83-a6eb-97812b694793\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" Oct 31 13:38:22.315091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.312743 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245qz\" (UniqueName: \"kubernetes.io/projected/df4f4e60-7bc8-4d83-a6eb-97812b694793-kube-api-access-245qz\") pod \"payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs\" (UID: \"df4f4e60-7bc8-4d83-a6eb-97812b694793\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" Oct 31 13:38:22.315091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.312804 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q\" (UID: \"402e298e-b8a6-4cef-8fa4-a5e667a33ca4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" Oct 31 13:38:22.315091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.312830 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgtj7\" (UniqueName: \"kubernetes.io/projected/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-kube-api-access-mgtj7\") pod \"payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q\" (UID: \"402e298e-b8a6-4cef-8fa4-a5e667a33ca4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" Oct 31 13:38:22.315091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.313705 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4f4e60-7bc8-4d83-a6eb-97812b694793-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs\" (UID: \"df4f4e60-7bc8-4d83-a6eb-97812b694793\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" Oct 31 13:38:22.315091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.314065 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q\" (UID: \"402e298e-b8a6-4cef-8fa4-a5e667a33ca4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" Oct 31 13:38:22.329349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.329303 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-245qz\" (UniqueName: \"kubernetes.io/projected/df4f4e60-7bc8-4d83-a6eb-97812b694793-kube-api-access-245qz\") pod \"payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs\" (UID: \"df4f4e60-7bc8-4d83-a6eb-97812b694793\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" Oct 31 13:38:22.330617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.330570 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgtj7\" (UniqueName: \"kubernetes.io/projected/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-kube-api-access-mgtj7\") pod \"payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q\" (UID: \"402e298e-b8a6-4cef-8fa4-a5e667a33ca4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" Oct 31 13:38:22.347242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.346958 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" Oct 31 13:38:22.562771 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.562685 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" Oct 31 13:38:22.671146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.671100 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q"] Oct 31 13:38:22.868703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:22.868675 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs"] Oct 31 13:38:22.869859 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:38:22.869825 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf4f4e60_7bc8_4d83_a6eb_97812b694793.slice/crio-2c76fd2aea568965cc498c87bbdb483a70518c2ca18118d458dd9c7517d38b5c WatchSource:0}: Error finding container 2c76fd2aea568965cc498c87bbdb483a70518c2ca18118d458dd9c7517d38b5c: Status 404 returned error can't find the container with id 2c76fd2aea568965cc498c87bbdb483a70518c2ca18118d458dd9c7517d38b5c Oct 31 13:38:23.102272 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.102187 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf"] Oct 31 13:38:23.107092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.107070 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" Oct 31 13:38:23.115034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.114988 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-4fde10\"" Oct 31 13:38:23.122404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.122379 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf"] Oct 31 13:38:23.129516 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.129459 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98klb\" (UniqueName: \"kubernetes.io/projected/f61f37d8-8b15-4955-bf9d-33a85b280110-kube-api-access-98klb\") pod \"payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf\" (UID: \"f61f37d8-8b15-4955-bf9d-33a85b280110\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" Oct 31 13:38:23.129852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.129821 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61f37d8-8b15-4955-bf9d-33a85b280110-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf\" (UID: \"f61f37d8-8b15-4955-bf9d-33a85b280110\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" Oct 31 13:38:23.151961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.151936 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb"] Oct 31 13:38:23.156647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.156624 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" Oct 31 13:38:23.166315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.166286 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-4fde10-dockercfg-46kfn\"" Oct 31 13:38:23.171402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.169305 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb"] Oct 31 13:38:23.235618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.233067 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61f37d8-8b15-4955-bf9d-33a85b280110-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf\" (UID: \"f61f37d8-8b15-4955-bf9d-33a85b280110\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" Oct 31 13:38:23.235618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.233269 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98klb\" (UniqueName: \"kubernetes.io/projected/f61f37d8-8b15-4955-bf9d-33a85b280110-kube-api-access-98klb\") pod \"payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf\" (UID: \"f61f37d8-8b15-4955-bf9d-33a85b280110\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" Oct 31 13:38:23.235618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.234628 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61f37d8-8b15-4955-bf9d-33a85b280110-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf\" (UID: \"f61f37d8-8b15-4955-bf9d-33a85b280110\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" Oct 31 13:38:23.324297 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.323629 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98klb\" (UniqueName: \"kubernetes.io/projected/f61f37d8-8b15-4955-bf9d-33a85b280110-kube-api-access-98klb\") pod \"payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf\" (UID: \"f61f37d8-8b15-4955-bf9d-33a85b280110\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" Oct 31 13:38:23.389810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.389694 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvt4x\" (UniqueName: \"kubernetes.io/projected/ad4ab732-0660-42a4-80e8-ebed759dd941-kube-api-access-lvt4x\") pod \"payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb\" (UID: \"ad4ab732-0660-42a4-80e8-ebed759dd941\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" Oct 31 13:38:23.413627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.411958 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4ab732-0660-42a4-80e8-ebed759dd941-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb\" (UID: \"ad4ab732-0660-42a4-80e8-ebed759dd941\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" Oct 31 13:38:23.429399 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.429359 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" Oct 31 13:38:23.451946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.450532 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" event={"ID":"402e298e-b8a6-4cef-8fa4-a5e667a33ca4","Type":"ContainerStarted","Data":"d82c143b76cf5752c264f05efa1a66bb7450be8cb88aa7be35e66401042f8f33"} Oct 31 13:38:23.451946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.450586 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" event={"ID":"402e298e-b8a6-4cef-8fa4-a5e667a33ca4","Type":"ContainerStarted","Data":"4876cd82a47517dacf1b6011c2dcca2e2796b4e63d61838cdfbd32dc8483c53b"} Oct 31 13:38:23.451946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.451625 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" Oct 31 13:38:23.453551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.453530 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" event={"ID":"df4f4e60-7bc8-4d83-a6eb-97812b694793","Type":"ContainerStarted","Data":"3236bb00f7e74a4b265cfacbd0f9ea3b2a71c52e143145e0843ab00f8bbc0619"} Oct 31 13:38:23.453756 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.453736 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" Oct 31 13:38:23.453839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.453770 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" event={"ID":"df4f4e60-7bc8-4d83-a6eb-97812b694793","Type":"ContainerStarted","Data":"2c76fd2aea568965cc498c87bbdb483a70518c2ca18118d458dd9c7517d38b5c"} Oct 31 13:38:23.482426 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.482218 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" podStartSLOduration=2.482192781 podStartE2EDuration="2.482192781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:23.473957404 +0000 UTC m=+2071.619701232" watchObservedRunningTime="2026-10-31 13:38:23.482192781 +0000 UTC m=+2071.627936615" Oct 31 13:38:23.495656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.495230 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" podStartSLOduration=2.495212462 podStartE2EDuration="2.495212462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:23.493469439 +0000 UTC m=+2071.639213272" watchObservedRunningTime="2026-10-31 13:38:23.495212462 +0000 UTC m=+2071.640956291" Oct 31 13:38:23.521236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.521198 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvt4x\" (UniqueName: \"kubernetes.io/projected/ad4ab732-0660-42a4-80e8-ebed759dd941-kube-api-access-lvt4x\") pod \"payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb\" (UID: \"ad4ab732-0660-42a4-80e8-ebed759dd941\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" Oct 31 13:38:23.521458 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.521441 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4ab732-0660-42a4-80e8-ebed759dd941-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb\" (UID: \"ad4ab732-0660-42a4-80e8-ebed759dd941\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" Oct 31 13:38:23.523294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.522394 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4ab732-0660-42a4-80e8-ebed759dd941-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb\" (UID: \"ad4ab732-0660-42a4-80e8-ebed759dd941\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" Oct 31 13:38:23.536283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.536194 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvt4x\" (UniqueName: \"kubernetes.io/projected/ad4ab732-0660-42a4-80e8-ebed759dd941-kube-api-access-lvt4x\") pod \"payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb\" (UID: \"ad4ab732-0660-42a4-80e8-ebed759dd941\") " pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" Oct 31 13:38:23.769235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.769199 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf"] Oct 31 13:38:23.769718 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:38:23.769691 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61f37d8_8b15_4955_bf9d_33a85b280110.slice/crio-740ace4e4b5b4982b612772cc12397763cb4411f61d8b21c78b02eda2d46a0e4 WatchSource:0}: Error finding container 740ace4e4b5b4982b612772cc12397763cb4411f61d8b21c78b02eda2d46a0e4: Status 404 returned error can't find the container with id 740ace4e4b5b4982b612772cc12397763cb4411f61d8b21c78b02eda2d46a0e4 Oct 31 13:38:23.779409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:23.779388 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" Oct 31 13:38:24.091280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:24.091248 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb"] Oct 31 13:38:24.091527 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:38:24.091503 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4ab732_0660_42a4_80e8_ebed759dd941.slice/crio-725bfc1f7ec756a773d7bf33eaa9e9a7321740c3a829768e7a07de951e1ac325 WatchSource:0}: Error finding container 725bfc1f7ec756a773d7bf33eaa9e9a7321740c3a829768e7a07de951e1ac325: Status 404 returned error can't find the container with id 725bfc1f7ec756a773d7bf33eaa9e9a7321740c3a829768e7a07de951e1ac325 Oct 31 13:38:24.464822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:24.464776 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" event={"ID":"f61f37d8-8b15-4955-bf9d-33a85b280110","Type":"ContainerStarted","Data":"77def19e6fb0460e6d6f6b79ede5dd3230e36d81693c041bde51d99e77f99b54"} Oct 31 13:38:24.465840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:24.464852 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" event={"ID":"f61f37d8-8b15-4955-bf9d-33a85b280110","Type":"ContainerStarted","Data":"740ace4e4b5b4982b612772cc12397763cb4411f61d8b21c78b02eda2d46a0e4"} Oct 31 13:38:24.465840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:24.464884 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" Oct 31 13:38:24.467149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:24.467118 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" event={"ID":"ad4ab732-0660-42a4-80e8-ebed759dd941","Type":"ContainerStarted","Data":"dfdc707f45b16b8d3e5df23baae013723448881a433a8e0242248cdc79613eef"} Oct 31 13:38:24.467299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:24.467177 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" event={"ID":"ad4ab732-0660-42a4-80e8-ebed759dd941","Type":"ContainerStarted","Data":"725bfc1f7ec756a773d7bf33eaa9e9a7321740c3a829768e7a07de951e1ac325"} Oct 31 13:38:24.467504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:24.467484 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" Oct 31 13:38:24.486022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:24.485973 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" podStartSLOduration=1.485960103 podStartE2EDuration="1.485960103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:24.485197574 +0000 UTC m=+2072.630941404" watchObservedRunningTime="2026-10-31 13:38:24.485960103 +0000 UTC m=+2072.631703930" Oct 31 13:38:24.506193 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:24.506143 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" podStartSLOduration=1.50612872 podStartE2EDuration="1.50612872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:24.504653578 +0000 UTC m=+2072.650397407" watchObservedRunningTime="2026-10-31 13:38:24.50612872 +0000 UTC m=+2072.651872549" Oct 31 13:38:27.787299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:27.787261 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr"] Oct 31 13:38:27.792784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:27.792760 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:27.817494 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:27.817469 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr"] Oct 31 13:38:27.874324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:27.874289 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f4232c-openshift-default-dockercfg-jrtvl\"" Oct 31 13:38:27.915653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:27.915579 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:27.915833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:27.915720 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:27.915833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:27.915745 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:27.915956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:27.915923 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:27.916015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:27.915989 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:27.916064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:27.916025 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:27.916168 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:27.916149 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:27.916269 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:27.916256 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:27.916320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:27.916291 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7f5m\" (UniqueName: \"kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-kube-api-access-n7f5m\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.017854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.017822 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.018048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.017907 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.018048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.017930 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.018048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.018004 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.018048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.018035 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.018296 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.018067 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.018296 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.018281 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.018392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.018297 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.018392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.018319 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.018392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.018360 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.018493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.018448 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.018493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.018486 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7f5m\" (UniqueName: \"kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-kube-api-access-n7f5m\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.018786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.018761 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.019184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.019160 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.020577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.020557 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.020759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.020740 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.033783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.033757 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.033999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.033979 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7f5m\" (UniqueName: \"kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-kube-api-access-n7f5m\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.104340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.104265 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" Oct 31 13:38:28.104755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.104733 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" Oct 31 13:38:28.115514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.115492 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:28.403196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.403154 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p"] Oct 31 13:38:28.403535 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.403477 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" podUID="8305b055-0baa-438e-b283-c4aca7c0926b" containerName="maas-api" containerID="cri-o://6cb755eb84e54fbf57fb94a451884ebde91277b7231a9317a4a91b4fc5b1f61d" gracePeriod=30 Oct 31 13:38:28.443186 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.443160 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr"] Oct 31 13:38:28.529780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.529738 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:38:28.529872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.529835 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:38:28.529918 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.529882 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:38:28.951621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:28.948275 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:38:29.083572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.049370 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8305b055-0baa-438e-b283-c4aca7c0926b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8305b055-0baa-438e-b283-c4aca7c0926b-openshift-service-ca\") pod \"8305b055-0baa-438e-b283-c4aca7c0926b\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " Oct 31 13:38:29.083572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.049437 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8305b055-0baa-438e-b283-c4aca7c0926b-kube-api-access-lhbzh\" (UniqueName: \"kubernetes.io/projected/8305b055-0baa-438e-b283-c4aca7c0926b-kube-api-access-lhbzh\") pod \"8305b055-0baa-438e-b283-c4aca7c0926b\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " Oct 31 13:38:29.083572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.049521 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-maas-api-tls\") pod \"8305b055-0baa-438e-b283-c4aca7c0926b\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " Oct 31 13:38:29.083572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.049726 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-metrics-certs\") pod \"8305b055-0baa-438e-b283-c4aca7c0926b\" (UID: \"8305b055-0baa-438e-b283-c4aca7c0926b\") " Oct 31 13:38:29.083572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.053512 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-metrics-certs" pod "8305b055-0baa-438e-b283-c4aca7c0926b" (UID: "8305b055-0baa-438e-b283-c4aca7c0926b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:38:29.083572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.055082 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8305b055-0baa-438e-b283-c4aca7c0926b-openshift-service-ca" pod "8305b055-0baa-438e-b283-c4aca7c0926b" (UID: "8305b055-0baa-438e-b283-c4aca7c0926b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:29.083572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.083424 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-maas-api-tls" pod "8305b055-0baa-438e-b283-c4aca7c0926b" (UID: "8305b055-0baa-438e-b283-c4aca7c0926b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:38:29.090945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.090895 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8305b055-0baa-438e-b283-c4aca7c0926b-kube-api-access-lhbzh" pod "8305b055-0baa-438e-b283-c4aca7c0926b" (UID: "8305b055-0baa-438e-b283-c4aca7c0926b"). InnerVolumeSpecName "kube-api-access-lhbzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:29.152493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.152454 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:29.152493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.152484 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8305b055-0baa-438e-b283-c4aca7c0926b-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:29.152493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.152497 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhbzh\" (UniqueName: \"kubernetes.io/projected/8305b055-0baa-438e-b283-c4aca7c0926b-kube-api-access-lhbzh\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:29.152745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.152513 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8305b055-0baa-438e-b283-c4aca7c0926b-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:29.544675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.543166 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" event={"ID":"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4","Type":"ContainerStarted","Data":"383c0bee28290db60b1efb038597fd9c398e22225df37ef48c6ac21dc379fb41"} Oct 31 13:38:29.544675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.543248 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" event={"ID":"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4","Type":"ContainerStarted","Data":"3e95b005030d3a4c53fab2fc86e5e144b47a40e3250ac9919ca9be82f0117f6a"} Oct 31 13:38:29.552689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.550545 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" event={"ID":"8305b055-0baa-438e-b283-c4aca7c0926b","Type":"ContainerDied","Data":"6cb755eb84e54fbf57fb94a451884ebde91277b7231a9317a4a91b4fc5b1f61d"} Oct 31 13:38:29.552689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.550648 5532 scope.go:122] "RemoveContainer" containerID="6cb755eb84e54fbf57fb94a451884ebde91277b7231a9317a4a91b4fc5b1f61d" Oct 31 13:38:29.552689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.550822 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" Oct 31 13:38:29.552689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.551393 5532 generic.go:356] "Generic (PLEG): container finished" podID="8305b055-0baa-438e-b283-c4aca7c0926b" containerID="6cb755eb84e54fbf57fb94a451884ebde91277b7231a9317a4a91b4fc5b1f61d" exitCode=0 Oct 31 13:38:29.552689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.551511 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p" event={"ID":"8305b055-0baa-438e-b283-c4aca7c0926b","Type":"ContainerDied","Data":"67d38ef7112685a66a41d7e6ccf2ade7ebbfc45a7fb3a1aa5b3c12a33a7aa8a3"} Oct 31 13:38:29.573057 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.573034 5532 scope.go:122] "RemoveContainer" containerID="6cb755eb84e54fbf57fb94a451884ebde91277b7231a9317a4a91b4fc5b1f61d" Oct 31 13:38:29.573360 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:38:29.573336 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb755eb84e54fbf57fb94a451884ebde91277b7231a9317a4a91b4fc5b1f61d\": container with ID starting with 6cb755eb84e54fbf57fb94a451884ebde91277b7231a9317a4a91b4fc5b1f61d not found: ID does not exist" containerID="6cb755eb84e54fbf57fb94a451884ebde91277b7231a9317a4a91b4fc5b1f61d" Oct 31 13:38:29.573450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.573368 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb755eb84e54fbf57fb94a451884ebde91277b7231a9317a4a91b4fc5b1f61d"} err="failed to get container status \"6cb755eb84e54fbf57fb94a451884ebde91277b7231a9317a4a91b4fc5b1f61d\": rpc error: code = NotFound desc = could not find container \"6cb755eb84e54fbf57fb94a451884ebde91277b7231a9317a4a91b4fc5b1f61d\": container with ID starting with 6cb755eb84e54fbf57fb94a451884ebde91277b7231a9317a4a91b4fc5b1f61d not found: ID does not exist" Oct 31 13:38:29.581050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.581002 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" podStartSLOduration=2.580986423 podStartE2EDuration="2.580986423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:29.579025404 +0000 UTC m=+2077.724769232" watchObservedRunningTime="2026-10-31 13:38:29.580986423 +0000 UTC m=+2077.726730250" Oct 31 13:38:29.598450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.598416 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p"] Oct 31 13:38:29.603256 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:29.603224 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cbb3d5-79c5bd5bf5-6wl9p"] Oct 31 13:38:30.010227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:30.010186 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8305b055-0baa-438e-b283-c4aca7c0926b" path="/var/lib/kubelet/pods/8305b055-0baa-438e-b283-c4aca7c0926b/volumes" Oct 31 13:38:30.116662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:30.116632 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:30.117831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:30.117805 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 31 13:38:30.117941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:30.117853 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" podUID="3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 31 13:38:31.116637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:31.116606 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 31 13:38:31.117136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:31.116672 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" podUID="3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 31 13:38:31.948618 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:38:31.948363 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:38:32.116650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:32.116380 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 31 13:38:32.116650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:32.116440 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" podUID="3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 31 13:38:33.116470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:33.116437 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 31 13:38:33.116711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:33.116492 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" podUID="3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 31 13:38:34.115927 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:34.115896 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 31 13:38:34.116121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:34.115945 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" podUID="3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 31 13:38:34.472473 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:34.472437 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" Oct 31 13:38:35.116242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:35.116204 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Oct 31 13:38:35.116430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:35.116258 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" podUID="3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Oct 31 13:38:35.489566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:35.489474 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" Oct 31 13:38:35.492195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:35.492161 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" Oct 31 13:38:35.494164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:35.494143 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" Oct 31 13:38:36.120752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:36.120721 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:36.121191 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:36.121173 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:36.121816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:36.121794 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:38:38.640621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:38.636536 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns"] Oct 31 13:38:38.640621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:38.637221 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" containerID="cri-o://7b13af30176895375fb1dc699eee8d87ec20f6cb3e2c55fce120c6bfc074846a" gracePeriod=30 Oct 31 13:38:40.154589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:40.153962 5532 patch_prober.go:29] interesting pod/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:38:40.154589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:40.154037 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:38:40.478924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:40.478768 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc"] Oct 31 13:38:40.483889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:40.483862 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-c45a37-b4xmc"] Oct 31 13:38:42.008249 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:42.008210 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1fc28fd-67d3-4e77-8555-9e51fb406358" path="/var/lib/kubelet/pods/f1fc28fd-67d3-4e77-8555-9e51fb406358/volumes" Oct 31 13:38:42.449875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:42.449840 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:38:44.116122 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.116097 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:38:44.143580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.143548 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-data\") pod \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " Oct 31 13:38:44.143815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.143792 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-certs\") pod \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " Oct 31 13:38:44.143896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.143853 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-token\" (UniqueName: \"kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-token\") pod \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " Oct 31 13:38:44.143977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.143964 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-socket\") pod \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " Oct 31 13:38:44.144117 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.144103 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-credential-socket\") pod \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " Oct 31 13:38:44.144180 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.144154 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-envoy\") pod \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " Oct 31 13:38:44.144275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.144256 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istiod-ca-cert\") pod \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " Oct 31 13:38:44.144552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.144530 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-kube-api-access-5dnbm\" (UniqueName: \"kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-kube-api-access-5dnbm\") pod \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " Oct 31 13:38:44.144752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.144734 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-podinfo\") pod \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\" (UID: \"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732\") " Oct 31 13:38:44.160799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.160361 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-certs" pod "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" (UID: "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:44.160799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.160734 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-data" pod "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" (UID: "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:44.161152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.161126 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istiod-ca-cert" pod "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" (UID: "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:38:44.161564 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.161543 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-socket" pod "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" (UID: "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:44.163968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.163941 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-token" pod "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" (UID: "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:44.164695 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.164671 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-envoy" pod "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" (UID: "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:44.164827 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.164732 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-kube-api-access-5dnbm" pod "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" (UID: "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732"). InnerVolumeSpecName "kube-api-access-5dnbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:38:44.165738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.165704 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-credential-socket" pod "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" (UID: "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:38:44.165831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.165792 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-podinfo" pod "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" (UID: "0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:38:44.253299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.253213 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dnbm\" (UniqueName: \"kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-kube-api-access-5dnbm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:44.253525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.253504 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:44.253652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.253639 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:44.253747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.253735 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:44.253834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.253823 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:44.253925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.253913 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:44.254009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.253997 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:44.254087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.254077 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:44.254160 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.254149 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:38:44.754682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.754641 5532 generic.go:356] "Generic (PLEG): container finished" podID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerID="7b13af30176895375fb1dc699eee8d87ec20f6cb3e2c55fce120c6bfc074846a" exitCode=0 Oct 31 13:38:44.754903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.754718 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" Oct 31 13:38:44.754903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.754723 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" event={"ID":"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732","Type":"ContainerDied","Data":"7b13af30176895375fb1dc699eee8d87ec20f6cb3e2c55fce120c6bfc074846a"} Oct 31 13:38:44.754903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.754836 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns" event={"ID":"0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732","Type":"ContainerDied","Data":"464d793504f8a98622bc68bb880a1c35b741fc1f18a19ae94d30849c4ba2ed76"} Oct 31 13:38:44.754903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.754862 5532 scope.go:122] "RemoveContainer" containerID="7b13af30176895375fb1dc699eee8d87ec20f6cb3e2c55fce120c6bfc074846a" Oct 31 13:38:44.767421 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.767401 5532 scope.go:122] "RemoveContainer" containerID="0927e77e1dc6ff013793181212c82b0b1d6ec76025504df6f05368c223e76448" Oct 31 13:38:44.779058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.779036 5532 scope.go:122] "RemoveContainer" containerID="7b13af30176895375fb1dc699eee8d87ec20f6cb3e2c55fce120c6bfc074846a" Oct 31 13:38:44.779648 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:38:44.779465 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b13af30176895375fb1dc699eee8d87ec20f6cb3e2c55fce120c6bfc074846a\": container with ID starting with 7b13af30176895375fb1dc699eee8d87ec20f6cb3e2c55fce120c6bfc074846a not found: ID does not exist" containerID="7b13af30176895375fb1dc699eee8d87ec20f6cb3e2c55fce120c6bfc074846a" Oct 31 13:38:44.779648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.779509 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b13af30176895375fb1dc699eee8d87ec20f6cb3e2c55fce120c6bfc074846a"} err="failed to get container status \"7b13af30176895375fb1dc699eee8d87ec20f6cb3e2c55fce120c6bfc074846a\": rpc error: code = NotFound desc = could not find container \"7b13af30176895375fb1dc699eee8d87ec20f6cb3e2c55fce120c6bfc074846a\": container with ID starting with 7b13af30176895375fb1dc699eee8d87ec20f6cb3e2c55fce120c6bfc074846a not found: ID does not exist" Oct 31 13:38:44.779648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.779538 5532 scope.go:122] "RemoveContainer" containerID="0927e77e1dc6ff013793181212c82b0b1d6ec76025504df6f05368c223e76448" Oct 31 13:38:44.780203 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:38:44.780174 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0927e77e1dc6ff013793181212c82b0b1d6ec76025504df6f05368c223e76448\": container with ID starting with 0927e77e1dc6ff013793181212c82b0b1d6ec76025504df6f05368c223e76448 not found: ID does not exist" containerID="0927e77e1dc6ff013793181212c82b0b1d6ec76025504df6f05368c223e76448" Oct 31 13:38:44.780306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.780213 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0927e77e1dc6ff013793181212c82b0b1d6ec76025504df6f05368c223e76448"} err="failed to get container status \"0927e77e1dc6ff013793181212c82b0b1d6ec76025504df6f05368c223e76448\": rpc error: code = NotFound desc = could not find container \"0927e77e1dc6ff013793181212c82b0b1d6ec76025504df6f05368c223e76448\": container with ID starting with 0927e77e1dc6ff013793181212c82b0b1d6ec76025504df6f05368c223e76448 not found: ID does not exist" Oct 31 13:38:44.789214 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.789188 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns"] Oct 31 13:38:44.795360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:44.795334 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-cbb3d5-openshift-default-69bf7c9485-wvzns"] Oct 31 13:38:45.589349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.589317 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh"] Oct 31 13:38:45.590547 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.590520 5532 cpu_manager.go:414] "containerMap: removing container" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" Oct 31 13:38:45.590547 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.590542 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" Oct 31 13:38:45.590816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.590630 5532 cpu_manager.go:414] "containerMap: removing container" podUID="8305b055-0baa-438e-b283-c4aca7c0926b" containerName="maas-api" Oct 31 13:38:45.590816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.590639 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8305b055-0baa-438e-b283-c4aca7c0926b" containerName="maas-api" Oct 31 13:38:45.590935 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.590842 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" Oct 31 13:38:45.590935 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.590857 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="8305b055-0baa-438e-b283-c4aca7c0926b" containerName="maas-api" Oct 31 13:38:45.596951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.596920 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:45.605850 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.605825 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-61f295-serving-cert\"" Oct 31 13:38:45.613047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.611244 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh"] Oct 31 13:38:45.693366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.693333 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-metrics-certs\") pod \"maas-api-e2e-inf-b-61f295-69ccb756-655hh\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:45.693573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.693490 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh4st\" (UniqueName: \"kubernetes.io/projected/f6165f0c-e157-4e98-9317-f0113e333f2e-kube-api-access-fh4st\") pod \"maas-api-e2e-inf-b-61f295-69ccb756-655hh\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:45.693713 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.693645 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-maas-api-tls\") pod \"maas-api-e2e-inf-b-61f295-69ccb756-655hh\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:45.693938 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.693911 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6165f0c-e157-4e98-9317-f0113e333f2e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61f295-69ccb756-655hh\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:45.795089 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.795058 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6165f0c-e157-4e98-9317-f0113e333f2e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61f295-69ccb756-655hh\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:45.795246 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.795154 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-metrics-certs\") pod \"maas-api-e2e-inf-b-61f295-69ccb756-655hh\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:45.795246 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.795228 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh4st\" (UniqueName: \"kubernetes.io/projected/f6165f0c-e157-4e98-9317-f0113e333f2e-kube-api-access-fh4st\") pod \"maas-api-e2e-inf-b-61f295-69ccb756-655hh\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:45.795348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.795319 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-maas-api-tls\") pod \"maas-api-e2e-inf-b-61f295-69ccb756-655hh\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:45.795809 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.795779 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6165f0c-e157-4e98-9317-f0113e333f2e-openshift-service-ca\") pod \"maas-api-e2e-inf-b-61f295-69ccb756-655hh\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:45.797775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.797746 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-maas-api-tls\") pod \"maas-api-e2e-inf-b-61f295-69ccb756-655hh\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:45.797775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.797761 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-metrics-certs\") pod \"maas-api-e2e-inf-b-61f295-69ccb756-655hh\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:45.809500 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.809474 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh4st\" (UniqueName: \"kubernetes.io/projected/f6165f0c-e157-4e98-9317-f0113e333f2e-kube-api-access-fh4st\") pod \"maas-api-e2e-inf-b-61f295-69ccb756-655hh\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:45.922245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:45.922212 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:46.017526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:46.016406 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" path="/var/lib/kubelet/pods/0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732/volumes" Oct 31 13:38:46.289215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:46.289182 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh"] Oct 31 13:38:46.290469 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:38:46.290427 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6165f0c_e157_4e98_9317_f0113e333f2e.slice/crio-6208eee197ffeffa02a5f13ce355468dc618c30ab40e1f04c67f4a5f24196238 WatchSource:0}: Error finding container 6208eee197ffeffa02a5f13ce355468dc618c30ab40e1f04c67f4a5f24196238: Status 404 returned error can't find the container with id 6208eee197ffeffa02a5f13ce355468dc618c30ab40e1f04c67f4a5f24196238 Oct 31 13:38:46.776155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:46.776123 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" event={"ID":"f6165f0c-e157-4e98-9317-f0113e333f2e","Type":"ContainerStarted","Data":"6208eee197ffeffa02a5f13ce355468dc618c30ab40e1f04c67f4a5f24196238"} Oct 31 13:38:47.787295 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:47.787257 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" event={"ID":"f6165f0c-e157-4e98-9317-f0113e333f2e","Type":"ContainerStarted","Data":"bc71d2a49a948d1a6286ae7591611a0d9979582ee25f6214501c794b38afab63"} Oct 31 13:38:47.787811 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:47.787323 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:47.817495 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:47.817438 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" podStartSLOduration=2.391931247 podStartE2EDuration="2.817422325s" totalImagesPullingTime="425.491078ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:47.815271235 +0000 UTC m=+2095.961015076" watchObservedRunningTime="2026-10-31 13:38:47.817422325 +0000 UTC m=+2095.963166180" Oct 31 13:38:51.229871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.229837 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl"] Oct 31 13:38:51.230921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.230896 5532 cpu_manager.go:414] "containerMap: removing container" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" Oct 31 13:38:51.230921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.230921 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" Oct 31 13:38:51.231188 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.231172 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1ac6be-f78a-4b20-9f5b-c8ac2a9d8732" containerName="istio-proxy" Oct 31 13:38:51.235802 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.235781 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" Oct 31 13:38:51.244034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.243832 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl"] Oct 31 13:38:51.245767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.245742 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0d835b\"" Oct 31 13:38:51.253935 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.253903 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0921726-c885-475d-9b36-0c899e6459e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl\" (UID: \"f0921726-c885-475d-9b36-0c899e6459e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" Oct 31 13:38:51.254105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.254084 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txn8m\" (UniqueName: \"kubernetes.io/projected/f0921726-c885-475d-9b36-0c899e6459e1-kube-api-access-txn8m\") pod \"payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl\" (UID: \"f0921726-c885-475d-9b36-0c899e6459e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" Oct 31 13:38:51.349750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.349703 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r"] Oct 31 13:38:51.363251 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.358675 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0921726-c885-475d-9b36-0c899e6459e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl\" (UID: \"f0921726-c885-475d-9b36-0c899e6459e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" Oct 31 13:38:51.363251 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.358758 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txn8m\" (UniqueName: \"kubernetes.io/projected/f0921726-c885-475d-9b36-0c899e6459e1-kube-api-access-txn8m\") pod \"payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl\" (UID: \"f0921726-c885-475d-9b36-0c899e6459e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" Oct 31 13:38:51.373232 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.371338 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0921726-c885-475d-9b36-0c899e6459e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl\" (UID: \"f0921726-c885-475d-9b36-0c899e6459e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" Oct 31 13:38:51.389139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.383134 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" Oct 31 13:38:51.389139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.388895 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r"] Oct 31 13:38:51.473401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.472958 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0d835b-dockercfg-kwzmm\"" Oct 31 13:38:51.476450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.476369 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txn8m\" (UniqueName: \"kubernetes.io/projected/f0921726-c885-475d-9b36-0c899e6459e1-kube-api-access-txn8m\") pod \"payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl\" (UID: \"f0921726-c885-475d-9b36-0c899e6459e1\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" Oct 31 13:38:51.592685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.592391 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7tk9\" (UniqueName: \"kubernetes.io/projected/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-kube-api-access-x7tk9\") pod \"payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r\" (UID: \"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" Oct 31 13:38:51.592685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.592535 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r\" (UID: \"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" Oct 31 13:38:51.612526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.593138 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" Oct 31 13:38:51.698186 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.698152 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7tk9\" (UniqueName: \"kubernetes.io/projected/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-kube-api-access-x7tk9\") pod \"payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r\" (UID: \"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" Oct 31 13:38:51.698337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.698280 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r\" (UID: \"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" Oct 31 13:38:51.699336 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.699310 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r\" (UID: \"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" Oct 31 13:38:51.723546 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.719801 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7tk9\" (UniqueName: \"kubernetes.io/projected/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-kube-api-access-x7tk9\") pod \"payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r\" (UID: \"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" Oct 31 13:38:51.782618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.778957 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" Oct 31 13:38:51.977118 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:51.977067 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl"] Oct 31 13:38:51.981472 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:38:51.981444 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0921726_c885_475d_9b36_0c899e6459e1.slice/crio-39d936dd905e9b28a8078ace7033b8e2e1a20ccfde12b606e862b06d1119461d WatchSource:0}: Error finding container 39d936dd905e9b28a8078ace7033b8e2e1a20ccfde12b606e862b06d1119461d: Status 404 returned error can't find the container with id 39d936dd905e9b28a8078ace7033b8e2e1a20ccfde12b606e862b06d1119461d Oct 31 13:38:52.140763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:52.140730 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r"] Oct 31 13:38:52.872933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:52.872878 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" event={"ID":"f0921726-c885-475d-9b36-0c899e6459e1","Type":"ContainerStarted","Data":"1b919f5f2c4a5960fc093f9affeca8f68894413aa51b026e57bb5c80ac40c553"} Oct 31 13:38:52.873468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:52.872961 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" event={"ID":"f0921726-c885-475d-9b36-0c899e6459e1","Type":"ContainerStarted","Data":"39d936dd905e9b28a8078ace7033b8e2e1a20ccfde12b606e862b06d1119461d"} Oct 31 13:38:52.874650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:52.874258 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" Oct 31 13:38:52.879840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:52.876632 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" event={"ID":"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab","Type":"ContainerStarted","Data":"a3a04168097d550b404fc8ce97fdd9b6aca56839ad87e4bb89aacd24a6faabc1"} Oct 31 13:38:52.879840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:52.876689 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" event={"ID":"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab","Type":"ContainerStarted","Data":"749ddfc2e4512420467bc2f77a21aa3cd08c9c290cce3d651670fcb6192df685"} Oct 31 13:38:52.880902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:52.880869 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" Oct 31 13:38:52.906333 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:52.906266 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" podStartSLOduration=1.906244979 podStartE2EDuration="1.906244979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:52.903023707 +0000 UTC m=+2101.048767547" watchObservedRunningTime="2026-10-31 13:38:52.906244979 +0000 UTC m=+2101.051988808" Oct 31 13:38:53.833467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:53.833366 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:38:53.866096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:53.866038 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" podStartSLOduration=2.8660233699999997 podStartE2EDuration="2.86602337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:52.941082254 +0000 UTC m=+2101.086826082" watchObservedRunningTime="2026-10-31 13:38:53.86602337 +0000 UTC m=+2102.011767199" Oct 31 13:38:54.636140 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:54.636105 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht"] Oct 31 13:38:54.642965 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:54.642936 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" Oct 31 13:38:54.649992 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:54.649966 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1ddade-dockercfg-xww8d\"" Oct 31 13:38:54.656841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:54.656814 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht"] Oct 31 13:38:54.688180 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:54.688137 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsm9d\" (UniqueName: \"kubernetes.io/projected/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-kube-api-access-dsm9d\") pod \"payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht\" (UID: \"30d3f968-575d-4bb8-8e8b-8918e9f03ed5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" Oct 31 13:38:54.688556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:54.688527 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht\" (UID: \"30d3f968-575d-4bb8-8e8b-8918e9f03ed5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" Oct 31 13:38:54.764619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:54.726897 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1ddade\"" Oct 31 13:38:54.833015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:54.812960 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsm9d\" (UniqueName: \"kubernetes.io/projected/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-kube-api-access-dsm9d\") pod \"payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht\" (UID: \"30d3f968-575d-4bb8-8e8b-8918e9f03ed5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" Oct 31 13:38:54.833015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:54.813112 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht\" (UID: \"30d3f968-575d-4bb8-8e8b-8918e9f03ed5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" Oct 31 13:38:54.833015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:54.814055 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht\" (UID: \"30d3f968-575d-4bb8-8e8b-8918e9f03ed5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" Oct 31 13:38:54.879676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:54.867146 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsm9d\" (UniqueName: \"kubernetes.io/projected/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-kube-api-access-dsm9d\") pod \"payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht\" (UID: \"30d3f968-575d-4bb8-8e8b-8918e9f03ed5\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" Oct 31 13:38:54.976446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:54.976414 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" Oct 31 13:38:55.388568 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:38:55.388538 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30d3f968_575d_4bb8_8e8b_8918e9f03ed5.slice/crio-b3dfda3253664cd41498c909225f2eca63ee161ca1d6729415a19df7095e7c80 WatchSource:0}: Error finding container b3dfda3253664cd41498c909225f2eca63ee161ca1d6729415a19df7095e7c80: Status 404 returned error can't find the container with id b3dfda3253664cd41498c909225f2eca63ee161ca1d6729415a19df7095e7c80 Oct 31 13:38:55.389575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:55.389550 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht"] Oct 31 13:38:55.976383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:55.976340 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" event={"ID":"30d3f968-575d-4bb8-8e8b-8918e9f03ed5","Type":"ContainerStarted","Data":"8218824cc36e51b2dbf919fecff6a9903dd70aa2f944acb5fec2873bab9feee8"} Oct 31 13:38:55.977243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:55.976408 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" event={"ID":"30d3f968-575d-4bb8-8e8b-8918e9f03ed5","Type":"ContainerStarted","Data":"b3dfda3253664cd41498c909225f2eca63ee161ca1d6729415a19df7095e7c80"} Oct 31 13:38:55.977243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:55.976442 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" Oct 31 13:38:56.003034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.002983 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" podStartSLOduration=2.002965596 podStartE2EDuration="2.002965596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:55.998810346 +0000 UTC m=+2104.144554177" watchObservedRunningTime="2026-10-31 13:38:56.002965596 +0000 UTC m=+2104.148709425" Oct 31 13:38:56.376304 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.376211 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r"] Oct 31 13:38:56.382579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.382308 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh"] Oct 31 13:38:56.382579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.382448 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.388544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.388515 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.406099 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.401756 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r"] Oct 31 13:38:56.407682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.407654 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh"] Oct 31 13:38:56.480133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.480107 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-61da9d-openshift-default-dockercfg-nzt6j\"" Oct 31 13:38:56.480954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.480931 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx5bv\" (UniqueName: \"kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-kube-api-access-wx5bv\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.481086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481069 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.481129 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481102 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.481129 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481124 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.481192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481140 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.481280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481261 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7327fd4-10e0-41ae-b540-03778f4df456-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.481374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481362 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d01fddda-bc47-40df-914b-d27a8363654e-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.481423 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481412 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.481459 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481438 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d01fddda-bc47-40df-914b-d27a8363654e-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.481490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481469 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.481524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481504 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7327fd4-10e0-41ae-b540-03778f4df456-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.481565 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481551 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.481639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481589 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5srcn\" (UniqueName: \"kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-kube-api-access-5srcn\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.481675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481666 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.481715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481696 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.481782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481771 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.481859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481848 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.481903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.481890 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.583352 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.583322 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5srcn\" (UniqueName: \"kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-kube-api-access-5srcn\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.583530 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.583373 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.583530 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.583419 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.583685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.583573 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.583748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.583728 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.583811 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.583797 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.583867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.583800 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.583867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.583853 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx5bv\" (UniqueName: \"kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-kube-api-access-wx5bv\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.583977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.583854 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.584038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.583980 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.584038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584011 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.584154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584041 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.584154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584083 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.584154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584086 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.584154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584150 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7327fd4-10e0-41ae-b540-03778f4df456-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.584356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584270 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d01fddda-bc47-40df-914b-d27a8363654e-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.584356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584295 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.584356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584332 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.584356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584352 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.584571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584397 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d01fddda-bc47-40df-914b-d27a8363654e-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.584571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584443 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.584571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584476 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7327fd4-10e0-41ae-b540-03778f4df456-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.584571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584479 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.584571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584506 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.584857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.584587 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.585092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.585067 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.585285 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.585249 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7327fd4-10e0-41ae-b540-03778f4df456-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.585634 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.585569 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d01fddda-bc47-40df-914b-d27a8363654e-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.586133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.586113 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.586354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.586334 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.586991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.586974 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d01fddda-bc47-40df-914b-d27a8363654e-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.587044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.587020 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7327fd4-10e0-41ae-b540-03778f4df456-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.600839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.600800 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5srcn\" (UniqueName: \"kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-kube-api-access-5srcn\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.602017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.601997 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:56.602837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.602816 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.602904 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.602824 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx5bv\" (UniqueName: \"kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-kube-api-access-wx5bv\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.724428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.723885 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:56.736867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:56.732937 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:57.067136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:57.067096 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r"] Oct 31 13:38:57.068392 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:38:57.068366 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7327fd4_10e0_41ae_b540_03778f4df456.slice/crio-400365d717315a12b3e7ba7a66d5591840a39e22c29eb4e3d2a9bd4347e30964 WatchSource:0}: Error finding container 400365d717315a12b3e7ba7a66d5591840a39e22c29eb4e3d2a9bd4347e30964: Status 404 returned error can't find the container with id 400365d717315a12b3e7ba7a66d5591840a39e22c29eb4e3d2a9bd4347e30964 Oct 31 13:38:57.073318 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:57.073281 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:38:57.073436 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:57.073357 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:38:57.073436 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:57.073385 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:38:57.088784 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:38:57.088750 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01fddda_bc47_40df_914b_d27a8363654e.slice/crio-7db4ad7efb9ad56b75b64d646f98b29fb0be561bc0b02694d98a9e97790c0c64 WatchSource:0}: Error finding container 7db4ad7efb9ad56b75b64d646f98b29fb0be561bc0b02694d98a9e97790c0c64: Status 404 returned error can't find the container with id 7db4ad7efb9ad56b75b64d646f98b29fb0be561bc0b02694d98a9e97790c0c64 Oct 31 13:38:57.090065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:57.090036 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh"] Oct 31 13:38:57.098002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:57.097947 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:38:57.098133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:57.098108 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:38:57.098242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:57.098222 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:38:57.900846 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:57.900806 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn"] Oct 31 13:38:57.909558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:57.909531 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:57.910477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:57.909423 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn"] Oct 31 13:38:58.018727 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.018688 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.018892 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.018762 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.018892 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.018818 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.022638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.019064 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.022638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.019142 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gzbc\" (UniqueName: \"kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-kube-api-access-2gzbc\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.022638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.019196 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.022638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.019352 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.022638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.019466 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.022638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.019538 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.022638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.020641 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" event={"ID":"d01fddda-bc47-40df-914b-d27a8363654e","Type":"ContainerStarted","Data":"e33bae3b70cfc93db66d404d8d84380adf900a5f4004acc52bc89f908a0a99b5"} Oct 31 13:38:58.022638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.020697 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" event={"ID":"d01fddda-bc47-40df-914b-d27a8363654e","Type":"ContainerStarted","Data":"7db4ad7efb9ad56b75b64d646f98b29fb0be561bc0b02694d98a9e97790c0c64"} Oct 31 13:38:58.023535 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.023506 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" event={"ID":"d7327fd4-10e0-41ae-b540-03778f4df456","Type":"ContainerStarted","Data":"1f4b5034ac08ec62ebde3d24c3813f8bc650dfdec2e70c0309144c1090285a19"} Oct 31 13:38:58.023651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.023558 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" event={"ID":"d7327fd4-10e0-41ae-b540-03778f4df456","Type":"ContainerStarted","Data":"400365d717315a12b3e7ba7a66d5591840a39e22c29eb4e3d2a9bd4347e30964"} Oct 31 13:38:58.114759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.114702 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" podStartSLOduration=2.114686589 podStartE2EDuration="2.114686589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:58.077349086 +0000 UTC m=+2106.223092915" watchObservedRunningTime="2026-10-31 13:38:58.114686589 +0000 UTC m=+2106.260430416" Oct 31 13:38:58.116135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.116098 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" podStartSLOduration=2.11608458 podStartE2EDuration="2.11608458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:38:58.115018949 +0000 UTC m=+2106.260762810" watchObservedRunningTime="2026-10-31 13:38:58.11608458 +0000 UTC m=+2106.261828411" Oct 31 13:38:58.120708 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.120673 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.120838 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.120751 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.120838 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.120778 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.120955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.120838 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.121111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.121093 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.121181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.121143 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-data\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.121277 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.121095 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-certs\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.121385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.121275 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-credential-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.121497 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.121304 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gzbc\" (UniqueName: \"kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-kube-api-access-2gzbc\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.121666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.121648 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.122078 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.121995 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istiod-ca-cert\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.122302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.122283 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.122553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.122533 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.122553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.122537 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-socket\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.124725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.124700 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-podinfo\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.128088 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.128066 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-envoy\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.136281 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.136260 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-token\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.136370 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.136280 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gzbc\" (UniqueName: \"kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-kube-api-access-2gzbc\") pod \"e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.299285 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.298945 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:38:58.724142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.724107 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:38:58.725363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.725340 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 31 13:38:58.725487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.725379 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" podUID="d7327fd4-10e0-41ae-b540-03778f4df456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 31 13:38:58.733871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.733802 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:38:58.736248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.735215 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 31 13:38:58.736248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.735278 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" podUID="d01fddda-bc47-40df-914b-d27a8363654e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 31 13:38:58.809019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.807547 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn"] Oct 31 13:38:58.815114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.815073 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:38:58.815217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.815183 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:38:58.815266 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.815235 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:38:58.825145 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.823717 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd"] Oct 31 13:38:58.833074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.832145 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd"] Oct 31 13:38:58.833074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.832261 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" Oct 31 13:38:58.836742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:58.836706 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-554759\"" Oct 31 13:38:59.219633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:59.211304 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klr22\" (UniqueName: \"kubernetes.io/projected/ed127dcb-547e-44e7-af88-c7b6ee439b81-kube-api-access-klr22\") pod \"payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd\" (UID: \"ed127dcb-547e-44e7-af88-c7b6ee439b81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" Oct 31 13:38:59.219633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:59.211362 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed127dcb-547e-44e7-af88-c7b6ee439b81-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd\" (UID: \"ed127dcb-547e-44e7-af88-c7b6ee439b81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" Oct 31 13:38:59.398369 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:59.396484 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klr22\" (UniqueName: \"kubernetes.io/projected/ed127dcb-547e-44e7-af88-c7b6ee439b81-kube-api-access-klr22\") pod \"payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd\" (UID: \"ed127dcb-547e-44e7-af88-c7b6ee439b81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" Oct 31 13:38:59.398369 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:59.396540 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed127dcb-547e-44e7-af88-c7b6ee439b81-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd\" (UID: \"ed127dcb-547e-44e7-af88-c7b6ee439b81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" Oct 31 13:38:59.422616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:59.407028 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed127dcb-547e-44e7-af88-c7b6ee439b81-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd\" (UID: \"ed127dcb-547e-44e7-af88-c7b6ee439b81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" Oct 31 13:38:59.422616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:59.419500 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klr22\" (UniqueName: \"kubernetes.io/projected/ed127dcb-547e-44e7-af88-c7b6ee439b81-kube-api-access-klr22\") pod \"payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd\" (UID: \"ed127dcb-547e-44e7-af88-c7b6ee439b81\") " pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" Oct 31 13:38:59.433651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:59.432831 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" event={"ID":"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6","Type":"ContainerStarted","Data":"75d8f78665d16da2f031877601d889c24a19507d40a46a34606bc946c5d15783"} Oct 31 13:38:59.517933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:59.517893 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" Oct 31 13:38:59.725772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:59.725728 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 31 13:38:59.726031 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:59.725998 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" podUID="d7327fd4-10e0-41ae-b540-03778f4df456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 31 13:38:59.733892 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:59.733860 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 31 13:38:59.734039 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:38:59.734024 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" podUID="d01fddda-bc47-40df-914b-d27a8363654e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 31 13:39:00.104031 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:00.104002 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd"] Oct 31 13:39:00.486620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:00.486083 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" event={"ID":"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6","Type":"ContainerStarted","Data":"accaf05a1efdd5fae8ed7ee16e88c0a011378cbe77b050d2069f3811a8a8fcc2"} Oct 31 13:39:00.491541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:00.491507 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" event={"ID":"ed127dcb-547e-44e7-af88-c7b6ee439b81","Type":"ContainerStarted","Data":"daf139136f7562240ea09369447cda0bb53cf17bc68ce4a026fe739e9a41a737"} Oct 31 13:39:00.491795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:00.491778 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" event={"ID":"ed127dcb-547e-44e7-af88-c7b6ee439b81","Type":"ContainerStarted","Data":"964e0954c25812bfe0d7bbe495e532eebaf0a2722c647fb525832d76298b6f22"} Oct 31 13:39:00.491894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:00.491884 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" Oct 31 13:39:00.519742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:00.519682 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" podStartSLOduration=3.519662716 podStartE2EDuration="3.519662716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:00.518188259 +0000 UTC m=+2108.663932091" watchObservedRunningTime="2026-10-31 13:39:00.519662716 +0000 UTC m=+2108.665406544" Oct 31 13:39:00.724999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:00.724958 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 31 13:39:00.725290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:00.725259 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" podUID="d7327fd4-10e0-41ae-b540-03778f4df456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 31 13:39:00.763904 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:00.763861 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 31 13:39:00.764215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:00.764180 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" podUID="d01fddda-bc47-40df-914b-d27a8363654e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 31 13:39:01.300419 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:01.300375 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:39:01.328812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:01.328765 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 31 13:39:01.330032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:01.329322 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" podUID="e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 31 13:39:01.724581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:01.724553 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 31 13:39:01.725028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:01.724913 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" podUID="d7327fd4-10e0-41ae-b540-03778f4df456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 31 13:39:01.733820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:01.733788 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 31 13:39:01.733820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:01.733831 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" podUID="d01fddda-bc47-40df-914b-d27a8363654e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 31 13:39:01.831332 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:39:01.831297 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:39:02.312621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:02.311776 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 31 13:39:02.312621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:02.311848 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" podUID="e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 31 13:39:02.737623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:02.733863 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 31 13:39:02.737623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:02.733925 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" podUID="d7327fd4-10e0-41ae-b540-03778f4df456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 31 13:39:02.764619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:02.758461 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 31 13:39:02.764619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:02.758521 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" podUID="d01fddda-bc47-40df-914b-d27a8363654e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 31 13:39:03.321173 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:03.321129 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 31 13:39:03.321357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:03.321194 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" podUID="e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 31 13:39:03.725681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:03.725173 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" start-of-body= Oct 31 13:39:03.725681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:03.725292 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" podUID="d7327fd4-10e0-41ae-b540-03778f4df456" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.174:15021/healthz/ready\": dial tcp 10.132.0.174:15021: connect: connection refused" Oct 31 13:39:03.733253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:03.733226 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" start-of-body= Oct 31 13:39:03.733359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:03.733280 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" podUID="d01fddda-bc47-40df-914b-d27a8363654e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.175:15021/healthz/ready\": dial tcp 10.132.0.175:15021: connect: connection refused" Oct 31 13:39:04.300314 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:04.300278 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 31 13:39:04.300716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:04.300339 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" podUID="e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 31 13:39:04.729184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:04.729153 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:39:04.729641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:04.729615 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:39:04.730245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:04.730222 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:39:04.738575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:04.738555 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:39:04.768684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:04.768649 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:39:04.769968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:04.769939 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:39:04.772805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:04.772763 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" podStartSLOduration=6.772751348 podStartE2EDuration="6.772751348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:00.575977745 +0000 UTC m=+2108.721721581" watchObservedRunningTime="2026-10-31 13:39:04.772751348 +0000 UTC m=+2112.918495176" Oct 31 13:39:04.978581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:04.978552 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" Oct 31 13:39:04.979908 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:04.979836 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" Oct 31 13:39:05.300084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:05.299986 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 31 13:39:05.300084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:05.300040 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" podUID="e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 31 13:39:06.300674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:06.300641 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" start-of-body= Oct 31 13:39:06.301158 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:06.300691 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" podUID="e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.176:15021/healthz/ready\": dial tcp 10.132.0.176:15021: connect: connection refused" Oct 31 13:39:07.015700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:07.015674 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" Oct 31 13:39:07.314627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:07.314515 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:39:07.315771 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:07.315746 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:39:07.317184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:07.317160 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:39:12.598761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:12.598728 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" Oct 31 13:39:31.764046 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:39:31.764007 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:39:41.496932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.496894 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz"] Oct 31 13:39:41.503678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.503653 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.511968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.511942 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-62778a\"/\"kube-root-ca.crt\"" Oct 31 13:39:41.512751 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.512724 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-62778a\"/\"openshift-service-ca.crt\"" Oct 31 13:39:41.512890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.512833 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-62778a\"/\"default-dockercfg-vqjw8\"" Oct 31 13:39:41.514553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.514529 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz"] Oct 31 13:39:41.514732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.514711 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-62778a\"/\"test-model-62778a-kserve-self-signed-certs\"" Oct 31 13:39:41.555499 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.555450 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-home\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.555754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.555585 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tmp-dir\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.555754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.555741 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-dshm\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.555948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.555930 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf278\" (UniqueName: \"kubernetes.io/projected/fb8e0c0b-e52a-4752-93e9-2624bbde361b-kube-api-access-lf278\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.556045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.556027 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-model-cache\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.556178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.556105 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tls-certs\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.657789 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.657749 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf278\" (UniqueName: \"kubernetes.io/projected/fb8e0c0b-e52a-4752-93e9-2624bbde361b-kube-api-access-lf278\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.658001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.657852 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-model-cache\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.658001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.657924 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tls-certs\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.658115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.658010 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-home\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.658115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.658092 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tmp-dir\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.658198 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.658171 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-dshm\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.658376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.658348 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-model-cache\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.658477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.658424 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-home\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.658477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.658452 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tmp-dir\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.660407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.660384 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-dshm\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.660505 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.660441 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tls-certs\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.692113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.692052 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf278\" (UniqueName: \"kubernetes.io/projected/fb8e0c0b-e52a-4752-93e9-2624bbde361b-kube-api-access-lf278\") pod \"test-model-62778a-kserve-78797649bf-957bz\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:41.825412 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:41.825314 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:42.216312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:42.216285 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz"] Oct 31 13:39:43.191008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:43.190974 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" event={"ID":"fb8e0c0b-e52a-4752-93e9-2624bbde361b","Type":"ContainerStarted","Data":"a4b10044b91d165dc532806c2759a338ecda5686db68afc939d4ebf767370b36"} Oct 31 13:39:43.191008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:43.191017 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" event={"ID":"fb8e0c0b-e52a-4752-93e9-2624bbde361b","Type":"ContainerStarted","Data":"975530d870a19e7f5549ebc1822ad07233fd894003e8d8aedf7a04f71769f2ff"} Oct 31 13:39:43.215139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:43.215069 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" podStartSLOduration=2.215049224 podStartE2EDuration="2.215049224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:43.214815225 +0000 UTC m=+2151.360559064" watchObservedRunningTime="2026-10-31 13:39:43.215049224 +0000 UTC m=+2151.360793055" Oct 31 13:39:43.826897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:43.826861 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:44.159001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.158964 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz"] Oct 31 13:39:44.165664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.165619 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.171697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.171671 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4fde10\"/\"kube-root-ca.crt\"" Oct 31 13:39:44.171857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.171717 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz"] Oct 31 13:39:44.172808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.172784 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-4fde10\"/\"openshift-service-ca.crt\"" Oct 31 13:39:44.172950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.172885 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4fde10\"/\"test-model-4fde10-kserve-self-signed-certs\"" Oct 31 13:39:44.173000 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.172955 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-4fde10\"/\"default-dockercfg-69tt7\"" Oct 31 13:39:44.257150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.257110 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-model-cache\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.257525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.257228 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-tmp-dir\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.257525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.257316 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkt4x\" (UniqueName: \"kubernetes.io/projected/8acd1643-d958-467c-8055-de8d1fff7ef2-kube-api-access-xkt4x\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.257525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.257433 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-home\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.257859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.257842 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-dshm\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.257905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.257896 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8acd1643-d958-467c-8055-de8d1fff7ef2-tls-certs\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.389617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.380673 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8acd1643-d958-467c-8055-de8d1fff7ef2-tls-certs\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.389617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.380846 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-model-cache\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.389617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.380911 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-tmp-dir\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.389617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.380969 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkt4x\" (UniqueName: \"kubernetes.io/projected/8acd1643-d958-467c-8055-de8d1fff7ef2-kube-api-access-xkt4x\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.389617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.381085 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-home\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.389617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.381320 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-dshm\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.389617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.382199 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-tmp-dir\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.389617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.383955 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-model-cache\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.389617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.384535 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-home\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.389617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.388141 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-dshm\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.412292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.412206 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkt4x\" (UniqueName: \"kubernetes.io/projected/8acd1643-d958-467c-8055-de8d1fff7ef2-kube-api-access-xkt4x\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.443665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.443580 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8acd1643-d958-467c-8055-de8d1fff7ef2-tls-certs\") pod \"test-model-4fde10-kserve-65cc657c67-7f8kz\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.486953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.486919 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:44.873261 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:39:44.873226 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8acd1643_d958_467c_8055_de8d1fff7ef2.slice/crio-49bbe8a732d27cbeaf195f45db7c835a81dca7b0cc8d762cc383052431f2e558 WatchSource:0}: Error finding container 49bbe8a732d27cbeaf195f45db7c835a81dca7b0cc8d762cc383052431f2e558: Status 404 returned error can't find the container with id 49bbe8a732d27cbeaf195f45db7c835a81dca7b0cc8d762cc383052431f2e558 Oct 31 13:39:44.873547 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:44.873530 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz"] Oct 31 13:39:45.239657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:45.238688 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" event={"ID":"8acd1643-d958-467c-8055-de8d1fff7ef2","Type":"ContainerStarted","Data":"970af36dd9ee84e8a39df9a594b3f451e6f5c5afc0159c9e3cb091d610a0724e"} Oct 31 13:39:45.239657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:45.238742 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" event={"ID":"8acd1643-d958-467c-8055-de8d1fff7ef2","Type":"ContainerStarted","Data":"49bbe8a732d27cbeaf195f45db7c835a81dca7b0cc8d762cc383052431f2e558"} Oct 31 13:39:45.487275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:45.487244 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:51.827440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:51.827403 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:51.840579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:51.840545 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:51.874441 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:51.874389 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" podStartSLOduration=7.87437296 podStartE2EDuration="7.87437296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:45.277900329 +0000 UTC m=+2153.423644158" watchObservedRunningTime="2026-10-31 13:39:51.87437296 +0000 UTC m=+2160.020116788" Oct 31 13:39:52.315407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:52.315377 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:39:54.488351 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:54.488317 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:54.502416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:54.502392 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:55.415568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:55.415533 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:39:56.530632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.530563 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj"] Oct 31 13:39:56.537583 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.537560 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.545950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.545918 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-61da9d\"/\"test-model-61da9d-kserve-self-signed-certs\"" Oct 31 13:39:56.546081 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.546045 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-61da9d\"/\"default-dockercfg-5hhk4\"" Oct 31 13:39:56.548915 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.548892 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj"] Oct 31 13:39:56.627980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.627937 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tmp-dir\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.628161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.628020 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-dshm\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.628161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.628128 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcrrs\" (UniqueName: \"kubernetes.io/projected/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-kube-api-access-vcrrs\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.628304 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.628282 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-home\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.628361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.628342 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tls-certs\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.628458 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.628446 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-model-cache\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.633220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.633193 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-61da9d\"/\"kube-root-ca.crt\"" Oct 31 13:39:56.633328 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.633251 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-61da9d\"/\"openshift-service-ca.crt\"" Oct 31 13:39:56.731644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.731560 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-model-cache\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.731889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.731867 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tmp-dir\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.731975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.731920 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-dshm\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.732045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.731979 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcrrs\" (UniqueName: \"kubernetes.io/projected/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-kube-api-access-vcrrs\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.732107 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.732035 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-model-cache\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.732190 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.732171 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-home\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.732246 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.732227 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tls-certs\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.732301 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.732243 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tmp-dir\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.732513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.732484 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-home\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.734403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.734378 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-dshm\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.735043 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.735025 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tls-certs\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.755288 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.755256 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcrrs\" (UniqueName: \"kubernetes.io/projected/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-kube-api-access-vcrrs\") pod \"test-model-61da9d-kserve-5fbddc95f7-pxsnj\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:56.853543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:56.853462 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:39:57.185340 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:39:57.185307 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9d69e89_8948_4a02_a8c5_35f1466b7a3f.slice/crio-484bc6fdf7f1b9f81e3a3bb0fcd7d1ed391673433a2b88d5820f89c33b854920 WatchSource:0}: Error finding container 484bc6fdf7f1b9f81e3a3bb0fcd7d1ed391673433a2b88d5820f89c33b854920: Status 404 returned error can't find the container with id 484bc6fdf7f1b9f81e3a3bb0fcd7d1ed391673433a2b88d5820f89c33b854920 Oct 31 13:39:57.188566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:57.188544 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj"] Oct 31 13:39:57.423074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:57.423035 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" event={"ID":"d9d69e89-8948-4a02-a8c5-35f1466b7a3f","Type":"ContainerStarted","Data":"5b6ba57eda2f2d2dd43387559fbdc832773a6f70269c9c71e094e86fa7797f22"} Oct 31 13:39:57.423074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:57.423078 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" event={"ID":"d9d69e89-8948-4a02-a8c5-35f1466b7a3f","Type":"ContainerStarted","Data":"484bc6fdf7f1b9f81e3a3bb0fcd7d1ed391673433a2b88d5820f89c33b854920"} Oct 31 13:39:57.451876 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:57.451784 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" podStartSLOduration=1.451770657 podStartE2EDuration="1.451770657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:39:57.448190535 +0000 UTC m=+2165.593934364" watchObservedRunningTime="2026-10-31 13:39:57.451770657 +0000 UTC m=+2165.597514484" Oct 31 13:39:57.853653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:39:57.853571 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:40:01.764221 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:40:01.764139 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:40:03.645947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:03.645916 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:40:06.854541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:06.854508 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:40:06.868804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:06.868775 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:40:07.583901 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:07.583868 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:40:31.763385 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:40:31.763347 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:40:45.593102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:45.593067 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk"] Oct 31 13:40:45.593728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:45.593267 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerName="istio-proxy" containerID="cri-o://45c723888cc9f8b5d600222bdaa5b1d81c1b30bdd8bf02fe07b6f4c12158d01b" gracePeriod=30 Oct 31 13:40:50.903416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.903386 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:40:50.971113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.970434 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-kube-api-access-9bpdg\" (UniqueName: \"kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-kube-api-access-9bpdg\") pod \"225e725e-7d21-4878-ac58-f9b9bb55f144\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " Oct 31 13:40:50.971113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.970497 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-podinfo\") pod \"225e725e-7d21-4878-ac58-f9b9bb55f144\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " Oct 31 13:40:50.971113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.970670 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-envoy\") pod \"225e725e-7d21-4878-ac58-f9b9bb55f144\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " Oct 31 13:40:50.971113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.970714 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225e725e-7d21-4878-ac58-f9b9bb55f144-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/225e725e-7d21-4878-ac58-f9b9bb55f144-istiod-ca-cert\") pod \"225e725e-7d21-4878-ac58-f9b9bb55f144\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " Oct 31 13:40:50.971113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.970879 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-token\" (UniqueName: \"kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-token\") pod \"225e725e-7d21-4878-ac58-f9b9bb55f144\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " Oct 31 13:40:50.974629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.973254 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-data\") pod \"225e725e-7d21-4878-ac58-f9b9bb55f144\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " Oct 31 13:40:50.974629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.973399 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-socket\") pod \"225e725e-7d21-4878-ac58-f9b9bb55f144\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " Oct 31 13:40:50.974629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.973496 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-certs\") pod \"225e725e-7d21-4878-ac58-f9b9bb55f144\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " Oct 31 13:40:50.974629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.973853 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-credential-socket\") pod \"225e725e-7d21-4878-ac58-f9b9bb55f144\" (UID: \"225e725e-7d21-4878-ac58-f9b9bb55f144\") " Oct 31 13:40:50.975511 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.975291 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225e725e-7d21-4878-ac58-f9b9bb55f144-istiod-ca-cert" pod "225e725e-7d21-4878-ac58-f9b9bb55f144" (UID: "225e725e-7d21-4878-ac58-f9b9bb55f144"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:40:50.976496 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.976177 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/225e725e-7d21-4878-ac58-f9b9bb55f144-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:50.976496 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.976448 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-credential-socket" pod "225e725e-7d21-4878-ac58-f9b9bb55f144" (UID: "225e725e-7d21-4878-ac58-f9b9bb55f144"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:50.977009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.976986 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-socket" pod "225e725e-7d21-4878-ac58-f9b9bb55f144" (UID: "225e725e-7d21-4878-ac58-f9b9bb55f144"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:50.977318 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.977297 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-data" pod "225e725e-7d21-4878-ac58-f9b9bb55f144" (UID: "225e725e-7d21-4878-ac58-f9b9bb55f144"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:50.977924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.977709 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-certs" pod "225e725e-7d21-4878-ac58-f9b9bb55f144" (UID: "225e725e-7d21-4878-ac58-f9b9bb55f144"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:50.979151 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.979074 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf"] Oct 31 13:40:50.982650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.981057 5532 cpu_manager.go:414] "containerMap: removing container" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerName="istio-proxy" Oct 31 13:40:50.982650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.981085 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerName="istio-proxy" Oct 31 13:40:50.982650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.981386 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerName="istio-proxy" Oct 31 13:40:50.984166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.984139 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-podinfo" pod "225e725e-7d21-4878-ac58-f9b9bb55f144" (UID: "225e725e-7d21-4878-ac58-f9b9bb55f144"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:40:51.004690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.989803 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-token" pod "225e725e-7d21-4878-ac58-f9b9bb55f144" (UID: "225e725e-7d21-4878-ac58-f9b9bb55f144"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:51.004690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.989915 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf"] Oct 31 13:40:51.004690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:50.990078 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.004690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.002588 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-cae9ec-openshift-default-dockercfg-pdrhx\"" Oct 31 13:40:51.005187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.005162 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-envoy" pod "225e725e-7d21-4878-ac58-f9b9bb55f144" (UID: "225e725e-7d21-4878-ac58-f9b9bb55f144"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:40:51.013057 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.013018 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-kube-api-access-9bpdg" pod "225e725e-7d21-4878-ac58-f9b9bb55f144" (UID: "225e725e-7d21-4878-ac58-f9b9bb55f144"). InnerVolumeSpecName "kube-api-access-9bpdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:40:51.124989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.111047 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:51.124989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.111095 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:51.124989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.111110 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:51.124989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.111124 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:51.124989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.111150 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:51.124989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.111163 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bpdg\" (UniqueName: \"kubernetes.io/projected/225e725e-7d21-4878-ac58-f9b9bb55f144-kube-api-access-9bpdg\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:51.124989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.111178 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:51.124989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.111190 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/225e725e-7d21-4878-ac58-f9b9bb55f144-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:40:51.136859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.136823 5532 generic.go:356] "Generic (PLEG): container finished" podID="225e725e-7d21-4878-ac58-f9b9bb55f144" containerID="45c723888cc9f8b5d600222bdaa5b1d81c1b30bdd8bf02fe07b6f4c12158d01b" exitCode=0 Oct 31 13:40:51.137166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.137142 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" event={"ID":"225e725e-7d21-4878-ac58-f9b9bb55f144","Type":"ContainerDied","Data":"45c723888cc9f8b5d600222bdaa5b1d81c1b30bdd8bf02fe07b6f4c12158d01b"} Oct 31 13:40:51.137300 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.137285 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" event={"ID":"225e725e-7d21-4878-ac58-f9b9bb55f144","Type":"ContainerDied","Data":"a73ce05f231d794f7b7346a0db44f90917842608b6ff859e6b7e33e1f671e627"} Oct 31 13:40:51.137393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.137383 5532 scope.go:122] "RemoveContainer" containerID="45c723888cc9f8b5d600222bdaa5b1d81c1b30bdd8bf02fe07b6f4c12158d01b" Oct 31 13:40:51.137524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.137392 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk" Oct 31 13:40:51.160944 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.160922 5532 scope.go:122] "RemoveContainer" containerID="45c723888cc9f8b5d600222bdaa5b1d81c1b30bdd8bf02fe07b6f4c12158d01b" Oct 31 13:40:51.161324 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:40:51.161302 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45c723888cc9f8b5d600222bdaa5b1d81c1b30bdd8bf02fe07b6f4c12158d01b\": container with ID starting with 45c723888cc9f8b5d600222bdaa5b1d81c1b30bdd8bf02fe07b6f4c12158d01b not found: ID does not exist" containerID="45c723888cc9f8b5d600222bdaa5b1d81c1b30bdd8bf02fe07b6f4c12158d01b" Oct 31 13:40:51.161408 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.161334 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45c723888cc9f8b5d600222bdaa5b1d81c1b30bdd8bf02fe07b6f4c12158d01b"} err="failed to get container status \"45c723888cc9f8b5d600222bdaa5b1d81c1b30bdd8bf02fe07b6f4c12158d01b\": rpc error: code = NotFound desc = could not find container \"45c723888cc9f8b5d600222bdaa5b1d81c1b30bdd8bf02fe07b6f4c12158d01b\": container with ID starting with 45c723888cc9f8b5d600222bdaa5b1d81c1b30bdd8bf02fe07b6f4c12158d01b not found: ID does not exist" Oct 31 13:40:51.178419 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.178389 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk"] Oct 31 13:40:51.184764 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.184734 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-pjjfk"] Oct 31 13:40:51.212873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.212839 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-envoy\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.213049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.212902 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-socket\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.213049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.212952 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-podinfo\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.213135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.213110 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-credential-socket\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.213178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.213150 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-certs\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.213220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.213211 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istiod-ca-cert\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.213297 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.213282 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2g6n\" (UniqueName: \"kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-kube-api-access-m2g6n\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.213347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.213335 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-token\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.213429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.213412 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-data\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.316181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.316145 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-data\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.316365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.316226 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-envoy\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.316365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.316281 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-socket\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.316365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.316297 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-podinfo\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.316524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.316442 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-credential-socket\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.316524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.316507 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-certs\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.316688 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.316561 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istiod-ca-cert\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.316688 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.316624 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-data\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.316688 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.316653 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2g6n\" (UniqueName: \"kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-kube-api-access-m2g6n\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.316861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.316731 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-token\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.316861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.316747 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-socket\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.316963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.316860 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-credential-socket\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.316963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.316893 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-certs\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.317373 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.317351 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istiod-ca-cert\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.318769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.318746 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-envoy\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.319009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.318992 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-podinfo\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.330390 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.330362 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-token\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.330651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.330586 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2g6n\" (UniqueName: \"kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-kube-api-access-m2g6n\") pod \"e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.437201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.437164 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:51.738064 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:40:51.738033 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ba4078_c9a3_4f92_933e_d6ef817c11aa.slice/crio-116931040d1f70539646b6012a34e04ea1ea3702408e057de6b8fa953469a528 WatchSource:0}: Error finding container 116931040d1f70539646b6012a34e04ea1ea3702408e057de6b8fa953469a528: Status 404 returned error can't find the container with id 116931040d1f70539646b6012a34e04ea1ea3702408e057de6b8fa953469a528 Oct 31 13:40:51.738567 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.738545 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf"] Oct 31 13:40:51.741579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.741547 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:40:51.741719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.741642 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:40:51.741719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:51.741674 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:40:52.041342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:52.041265 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="225e725e-7d21-4878-ac58-f9b9bb55f144" path="/var/lib/kubelet/pods/225e725e-7d21-4878-ac58-f9b9bb55f144/volumes" Oct 31 13:40:52.148732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:52.148701 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" event={"ID":"e5ba4078-c9a3-4f92-933e-d6ef817c11aa","Type":"ContainerStarted","Data":"147f136c4404154dbc6098038e91e712e05334656abf72cda91d3747e503149b"} Oct 31 13:40:52.148899 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:52.148751 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" event={"ID":"e5ba4078-c9a3-4f92-933e-d6ef817c11aa","Type":"ContainerStarted","Data":"116931040d1f70539646b6012a34e04ea1ea3702408e057de6b8fa953469a528"} Oct 31 13:40:52.180096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:52.180029 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" podStartSLOduration=13.180008416 podStartE2EDuration="13.180008416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:40:52.176503003 +0000 UTC m=+2220.322246831" watchObservedRunningTime="2026-10-31 13:40:52.180008416 +0000 UTC m=+2220.325752244" Oct 31 13:40:52.438294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:52.438247 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:52.440376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:52.440345 5532 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 31 13:40:52.440680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:52.440417 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" podUID="e5ba4078-c9a3-4f92-933e-d6ef817c11aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 31 13:40:53.437744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:53.437710 5532 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 31 13:40:53.438146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:53.437773 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" podUID="e5ba4078-c9a3-4f92-933e-d6ef817c11aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 31 13:40:53.867681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:53.867583 5532 scope.go:122] "RemoveContainer" containerID="c954889247eb06fdef42e59d0cc0f046a16486f992d1e367131a8739ebcc81d0" Oct 31 13:40:54.437589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:54.437555 5532 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 31 13:40:54.437816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:54.437671 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" podUID="e5ba4078-c9a3-4f92-933e-d6ef817c11aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 31 13:40:55.437501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:55.437465 5532 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 31 13:40:55.437703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:55.437516 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" podUID="e5ba4078-c9a3-4f92-933e-d6ef817c11aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 31 13:40:56.169803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.169758 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44"] Oct 31 13:40:56.176852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.176826 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" Oct 31 13:40:56.184881 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.184851 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44"] Oct 31 13:40:56.186497 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.186471 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-35b584-dockercfg-52r7x\"" Oct 31 13:40:56.199190 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.198784 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-35b584\"" Oct 31 13:40:56.212387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.212357 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8"] Oct 31 13:40:56.220304 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.220280 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" Oct 31 13:40:56.227613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.227571 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8"] Oct 31 13:40:56.325625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.321543 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwsnm\" (UniqueName: \"kubernetes.io/projected/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-kube-api-access-jwsnm\") pod \"payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8\" (UID: \"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" Oct 31 13:40:56.325625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.321679 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44\" (UID: \"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" Oct 31 13:40:56.325625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.321838 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8\" (UID: \"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" Oct 31 13:40:56.325625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.321924 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66tht\" (UniqueName: \"kubernetes.io/projected/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-kube-api-access-66tht\") pod \"payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44\" (UID: \"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" Oct 31 13:40:56.492687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.460230 5532 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 31 13:40:56.636733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.636688 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" podUID="e5ba4078-c9a3-4f92-933e-d6ef817c11aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 31 13:40:56.636977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.573501 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwsnm\" (UniqueName: \"kubernetes.io/projected/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-kube-api-access-jwsnm\") pod \"payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8\" (UID: \"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" Oct 31 13:40:56.637091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.482059 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwsnm\" (UniqueName: \"kubernetes.io/projected/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-kube-api-access-jwsnm\") pod \"payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8\" (UID: \"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" Oct 31 13:40:56.637432 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.637406 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44\" (UID: \"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" Oct 31 13:40:56.637749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.637731 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8\" (UID: \"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" Oct 31 13:40:56.637956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.637940 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66tht\" (UniqueName: \"kubernetes.io/projected/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-kube-api-access-66tht\") pod \"payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44\" (UID: \"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" Oct 31 13:40:56.682618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.639316 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8\" (UID: \"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5\") " pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" Oct 31 13:40:56.703618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.700231 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44\" (UID: \"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" Oct 31 13:40:56.729616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.710127 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" Oct 31 13:40:56.741169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.740623 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66tht\" (UniqueName: \"kubernetes.io/projected/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-kube-api-access-66tht\") pod \"payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44\" (UID: \"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" Oct 31 13:40:56.804135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:56.798062 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" Oct 31 13:40:57.275129 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:40:57.275096 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1bbc81_7ef4_497e_b4aa_e06b3db670f5.slice/crio-4e78b901de1e6c6bfb17c124d24ba693a8b69f198cb8932506c0127f3e5a5452 WatchSource:0}: Error finding container 4e78b901de1e6c6bfb17c124d24ba693a8b69f198cb8932506c0127f3e5a5452: Status 404 returned error can't find the container with id 4e78b901de1e6c6bfb17c124d24ba693a8b69f198cb8932506c0127f3e5a5452 Oct 31 13:40:57.275800 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:57.275752 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8"] Oct 31 13:40:57.279668 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:40:57.279641 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64e27b8e_58f0_41e7_a8e4_eb7c403b5b2c.slice/crio-6e27ad8d6d15fefc848c7bccb86236f7dc70217155a5fe58cb94926f83e2c8fb WatchSource:0}: Error finding container 6e27ad8d6d15fefc848c7bccb86236f7dc70217155a5fe58cb94926f83e2c8fb: Status 404 returned error can't find the container with id 6e27ad8d6d15fefc848c7bccb86236f7dc70217155a5fe58cb94926f83e2c8fb Oct 31 13:40:57.281020 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:57.280996 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44"] Oct 31 13:40:57.437668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:57.437620 5532 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 31 13:40:57.437836 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:57.437681 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" podUID="e5ba4078-c9a3-4f92-933e-d6ef817c11aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 31 13:40:58.227608 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:58.227557 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" event={"ID":"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c","Type":"ContainerStarted","Data":"641c80cb5498a5d776bdcf2047ea39fb9f542950256064964f6a451047a139ed"} Oct 31 13:40:58.227806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:58.227643 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" Oct 31 13:40:58.227806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:58.227673 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" event={"ID":"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c","Type":"ContainerStarted","Data":"6e27ad8d6d15fefc848c7bccb86236f7dc70217155a5fe58cb94926f83e2c8fb"} Oct 31 13:40:58.229391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:58.229366 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" event={"ID":"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5","Type":"ContainerStarted","Data":"ddeeceeac7253ba5ab475834d9721a9cc3b33ffbecf9788b7ad2b42fb21bcfb8"} Oct 31 13:40:58.229479 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:58.229408 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" event={"ID":"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5","Type":"ContainerStarted","Data":"4e78b901de1e6c6bfb17c124d24ba693a8b69f198cb8932506c0127f3e5a5452"} Oct 31 13:40:58.229479 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:58.229452 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" Oct 31 13:40:58.264622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:58.263758 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" podStartSLOduration=2.263736231 podStartE2EDuration="2.263736231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:40:58.260075232 +0000 UTC m=+2226.405819062" watchObservedRunningTime="2026-10-31 13:40:58.263736231 +0000 UTC m=+2226.409480084" Oct 31 13:40:58.295176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:58.295115 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" podStartSLOduration=2.295096386 podStartE2EDuration="2.295096386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:40:58.292629554 +0000 UTC m=+2226.438373381" watchObservedRunningTime="2026-10-31 13:40:58.295096386 +0000 UTC m=+2226.440840215" Oct 31 13:40:58.438586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:58.438556 5532 patch_prober.go:29] interesting pod/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" start-of-body= Oct 31 13:40:58.438955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:58.438919 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" podUID="e5ba4078-c9a3-4f92-933e-d6ef817c11aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.181:15021/healthz/ready\": dial tcp 10.132.0.181:15021: connect: connection refused" Oct 31 13:40:59.443103 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:59.443068 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:59.443565 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:59.443551 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:40:59.444349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:40:59.444330 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:41:01.764777 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:41:01.764742 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:41:09.242261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:09.242225 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" Oct 31 13:41:09.242772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:09.242746 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" Oct 31 13:41:30.678546 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:30.678439 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t"] Oct 31 13:41:30.679141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:30.678666 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" podUID="29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" containerName="istio-proxy" containerID="cri-o://786a4572ea4d476e76dc01c7664bf4201c7aaf018646b40999b52b7526750407" gracePeriod=30 Oct 31 13:41:31.763255 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:41:31.763219 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:41:35.979050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:35.979017 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:41:36.020794 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.020758 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4"] Oct 31 13:41:36.022184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.022156 5532 cpu_manager.go:414] "containerMap: removing container" podUID="29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" containerName="istio-proxy" Oct 31 13:41:36.022184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.022184 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" containerName="istio-proxy" Oct 31 13:41:36.022504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.022489 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" containerName="istio-proxy" Oct 31 13:41:36.027194 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.027170 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.041365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.041341 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4"] Oct 31 13:41:36.088983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.088946 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-certs\") pod \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " Oct 31 13:41:36.089192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.089009 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-data\") pod \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " Oct 31 13:41:36.089192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.089048 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-credential-socket\") pod \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " Oct 31 13:41:36.089192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.089114 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-socket\") pod \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " Oct 31 13:41:36.089192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.089185 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-podinfo\") pod \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " Oct 31 13:41:36.089429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.089335 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-token\" (UniqueName: \"kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-token\") pod \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " Oct 31 13:41:36.089429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.089342 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-credential-socket" pod "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" (UID: "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:36.089429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.089376 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istiod-ca-cert\") pod \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " Oct 31 13:41:36.089621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.089432 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-data" pod "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" (UID: "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:36.089690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.089654 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-envoy\") pod \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " Oct 31 13:41:36.089749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.089706 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-kube-api-access-27hd2\" (UniqueName: \"kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-kube-api-access-27hd2\") pod \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\" (UID: \"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc\") " Oct 31 13:41:36.089814 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.089761 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istiod-ca-cert" pod "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" (UID: "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:41:36.090161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.090130 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.090263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.090237 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-certs" pod "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" (UID: "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:36.090490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.090464 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkm4g\" (UniqueName: \"kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-kube-api-access-fkm4g\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.090661 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.090631 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27643442-2613-43f3-a046-4a3e8dcd56db-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.090765 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.090719 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.090765 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.090742 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-socket" pod "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" (UID: "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:36.090765 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.090760 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.091091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.091068 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.091155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.091128 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27643442-2613-43f3-a046-4a3e8dcd56db-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.091215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.091166 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.091314 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.091288 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.091460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.091440 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:36.091531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.091468 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:36.091531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.091482 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:36.091531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.091496 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:36.091531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.091511 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:36.094004 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.093975 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-token" pod "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" (UID: "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:36.094004 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.093990 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-podinfo" pod "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" (UID: "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:41:36.094179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.094020 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-kube-api-access-27hd2" pod "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" (UID: "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc"). InnerVolumeSpecName "kube-api-access-27hd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:36.095207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.095182 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-envoy" pod "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" (UID: "29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:36.192888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.192848 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.193092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.192912 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.193092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193036 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.193092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193073 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27643442-2613-43f3-a046-4a3e8dcd56db-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.193269 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193109 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.193269 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193163 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.193269 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193247 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.193419 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193311 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.193419 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193329 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.193537 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193413 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.193614 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193556 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkm4g\" (UniqueName: \"kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-kube-api-access-fkm4g\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.193745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193723 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27643442-2613-43f3-a046-4a3e8dcd56db-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.193839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193747 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.193944 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193928 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:36.194007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193953 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:36.194007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193969 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:36.194007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.193985 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27hd2\" (UniqueName: \"kubernetes.io/projected/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc-kube-api-access-27hd2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:36.194354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.194335 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27643442-2613-43f3-a046-4a3e8dcd56db-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.195501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.195477 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.195668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.195647 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27643442-2613-43f3-a046-4a3e8dcd56db-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.206253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.206204 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.206369 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.206351 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkm4g\" (UniqueName: \"kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-kube-api-access-fkm4g\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.357052 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.357022 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:36.669905 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:41:36.669874 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27643442_2613_43f3_a046_4a3e8dcd56db.slice/crio-e212db545401692d8a67799d882de69d617dbed27c956677bb50257d5b34fd74 WatchSource:0}: Error finding container e212db545401692d8a67799d882de69d617dbed27c956677bb50257d5b34fd74: Status 404 returned error can't find the container with id e212db545401692d8a67799d882de69d617dbed27c956677bb50257d5b34fd74 Oct 31 13:41:36.670174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.670152 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4"] Oct 31 13:41:36.672907 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.672877 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:41:36.672980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.672945 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:41:36.672980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.672973 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:41:36.699236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.699205 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" event={"ID":"27643442-2613-43f3-a046-4a3e8dcd56db","Type":"ContainerStarted","Data":"e212db545401692d8a67799d882de69d617dbed27c956677bb50257d5b34fd74"} Oct 31 13:41:36.700863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.700840 5532 generic.go:356] "Generic (PLEG): container finished" podID="29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" containerID="786a4572ea4d476e76dc01c7664bf4201c7aaf018646b40999b52b7526750407" exitCode=0 Oct 31 13:41:36.701664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.700910 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" event={"ID":"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc","Type":"ContainerDied","Data":"786a4572ea4d476e76dc01c7664bf4201c7aaf018646b40999b52b7526750407"} Oct 31 13:41:36.701664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.700941 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" event={"ID":"29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc","Type":"ContainerDied","Data":"fcc4cff8634c8a4d6b95c1a2b974aef6f5da4f308aea059ebb7a9cc8b8333532"} Oct 31 13:41:36.701664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.700965 5532 scope.go:122] "RemoveContainer" containerID="786a4572ea4d476e76dc01c7664bf4201c7aaf018646b40999b52b7526750407" Oct 31 13:41:36.701664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.701063 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t" Oct 31 13:41:36.729312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.729295 5532 scope.go:122] "RemoveContainer" containerID="786a4572ea4d476e76dc01c7664bf4201c7aaf018646b40999b52b7526750407" Oct 31 13:41:36.729580 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:41:36.729561 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786a4572ea4d476e76dc01c7664bf4201c7aaf018646b40999b52b7526750407\": container with ID starting with 786a4572ea4d476e76dc01c7664bf4201c7aaf018646b40999b52b7526750407 not found: ID does not exist" containerID="786a4572ea4d476e76dc01c7664bf4201c7aaf018646b40999b52b7526750407" Oct 31 13:41:36.729656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.729607 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786a4572ea4d476e76dc01c7664bf4201c7aaf018646b40999b52b7526750407"} err="failed to get container status \"786a4572ea4d476e76dc01c7664bf4201c7aaf018646b40999b52b7526750407\": rpc error: code = NotFound desc = could not find container \"786a4572ea4d476e76dc01c7664bf4201c7aaf018646b40999b52b7526750407\": container with ID starting with 786a4572ea4d476e76dc01c7664bf4201c7aaf018646b40999b52b7526750407 not found: ID does not exist" Oct 31 13:41:36.761567 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.761539 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t"] Oct 31 13:41:36.766273 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:36.766249 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-8t82t"] Oct 31 13:41:37.712531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:37.712494 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" event={"ID":"27643442-2613-43f3-a046-4a3e8dcd56db","Type":"ContainerStarted","Data":"fb1ef483cd12f6601133838a5ae08a190b326d6a978a9a4edd08a7f7c1185cf9"} Oct 31 13:41:37.765616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:37.765146 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" podStartSLOduration=131.765121986 podStartE2EDuration="2m11.765121986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:41:37.738249748 +0000 UTC m=+2265.883993577" watchObservedRunningTime="2026-10-31 13:41:37.765121986 +0000 UTC m=+2265.910865815" Oct 31 13:41:38.017684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:38.017586 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc" path="/var/lib/kubelet/pods/29ed7132-d493-4fc3-bc3b-b9e21fb3ebcc/volumes" Oct 31 13:41:38.363679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:38.358658 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:38.363679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:38.362689 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:41:38.363679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:38.362778 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" podUID="27643442-2613-43f3-a046-4a3e8dcd56db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 13:41:38.843619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:38.838567 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:41:39.360356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:39.360321 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:41:39.360649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:39.360620 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" podUID="27643442-2613-43f3-a046-4a3e8dcd56db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 13:41:40.363621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:40.360965 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:41:40.363621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:40.361028 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" podUID="27643442-2613-43f3-a046-4a3e8dcd56db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 13:41:41.363618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:41.362948 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:41:41.363618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:41.363012 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" podUID="27643442-2613-43f3-a046-4a3e8dcd56db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 13:41:42.363950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:42.363916 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:41:42.364354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:42.363974 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" podUID="27643442-2613-43f3-a046-4a3e8dcd56db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 13:41:43.357659 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:43.357403 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" start-of-body= Oct 31 13:41:43.357659 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:43.357471 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" podUID="27643442-2613-43f3-a046-4a3e8dcd56db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.184:15021/healthz/ready\": dial tcp 10.132.0.184:15021: connect: connection refused" Oct 31 13:41:44.362829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:44.362794 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:44.363278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:44.363262 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:44.363863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:44.363839 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:41:45.931469 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:45.931434 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc"] Oct 31 13:41:45.931977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:45.931731 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" podUID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" containerName="istio-proxy" containerID="cri-o://e71e671ac5d840b7f6ef1b60dfa465bbf836bbfc201d6a62984d1c04851f1017" gracePeriod=30 Oct 31 13:41:46.088240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:46.088206 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t"] Oct 31 13:41:46.088434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:46.088409 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" podUID="0a27b459-9ad5-4ade-87f3-6ef3ef735e30" containerName="istio-proxy" containerID="cri-o://1ac4a737aaae7c4b67f8dd0ec04ea5c623b623e7db8fbe9f836bfbdd80c25487" gracePeriod=30 Oct 31 13:41:51.241940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.241913 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:41:51.289048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.289017 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff"] Oct 31 13:41:51.290181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.290157 5532 cpu_manager.go:414] "containerMap: removing container" podUID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" containerName="istio-proxy" Oct 31 13:41:51.290289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.290183 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" containerName="istio-proxy" Oct 31 13:41:51.290559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.290521 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" containerName="istio-proxy" Oct 31 13:41:51.296699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.296659 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.330754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.330722 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff"] Oct 31 13:41:51.416890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.416295 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-kube-api-access-dqrmh\" (UniqueName: \"kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-kube-api-access-dqrmh\") pod \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " Oct 31 13:41:51.416890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.416463 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-token\" (UniqueName: \"kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-token\") pod \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " Oct 31 13:41:51.416890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.416538 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istiod-ca-cert\") pod \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " Oct 31 13:41:51.416890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.416670 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-envoy\") pod \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " Oct 31 13:41:51.416890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.416705 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-socket\") pod \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " Oct 31 13:41:51.416890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.416748 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-certs\") pod \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " Oct 31 13:41:51.416890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.416864 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-data\") pod \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " Oct 31 13:41:51.417358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.417063 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-credential-socket\") pod \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " Oct 31 13:41:51.417358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.417089 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-podinfo\") pod \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\" (UID: \"226dd9d4-80bc-4c59-8f41-d1cefc341cf9\") " Oct 31 13:41:51.417358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.417130 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istiod-ca-cert" pod "226dd9d4-80bc-4c59-8f41-d1cefc341cf9" (UID: "226dd9d4-80bc-4c59-8f41-d1cefc341cf9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:41:51.417506 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.417383 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrkbd\" (UniqueName: \"kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-kube-api-access-mrkbd\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.417506 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.417465 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.417624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.417535 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.417624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.417560 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.417731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.417669 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.417787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.417723 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-credential-socket" pod "226dd9d4-80bc-4c59-8f41-d1cefc341cf9" (UID: "226dd9d4-80bc-4c59-8f41-d1cefc341cf9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:51.418134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.417981 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-data" pod "226dd9d4-80bc-4c59-8f41-d1cefc341cf9" (UID: "226dd9d4-80bc-4c59-8f41-d1cefc341cf9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:51.418134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.418076 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.418528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.418204 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.418528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.418346 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.418672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.418621 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.418824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.418803 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.418901 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.418833 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.418901 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.418850 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.419002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.418962 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-certs" pod "226dd9d4-80bc-4c59-8f41-d1cefc341cf9" (UID: "226dd9d4-80bc-4c59-8f41-d1cefc341cf9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:51.419813 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.419716 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-socket" pod "226dd9d4-80bc-4c59-8f41-d1cefc341cf9" (UID: "226dd9d4-80bc-4c59-8f41-d1cefc341cf9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:51.420292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.420258 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-podinfo" pod "226dd9d4-80bc-4c59-8f41-d1cefc341cf9" (UID: "226dd9d4-80bc-4c59-8f41-d1cefc341cf9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:41:51.427886 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.427863 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:41:51.432158 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.432124 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-kube-api-access-dqrmh" pod "226dd9d4-80bc-4c59-8f41-d1cefc341cf9" (UID: "226dd9d4-80bc-4c59-8f41-d1cefc341cf9"). InnerVolumeSpecName "kube-api-access-dqrmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:51.433517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.433496 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-token" pod "226dd9d4-80bc-4c59-8f41-d1cefc341cf9" (UID: "226dd9d4-80bc-4c59-8f41-d1cefc341cf9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:51.433673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.433525 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-envoy" pod "226dd9d4-80bc-4c59-8f41-d1cefc341cf9" (UID: "226dd9d4-80bc-4c59-8f41-d1cefc341cf9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:51.470235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.470163 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng"] Oct 31 13:41:51.471144 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.471127 5532 cpu_manager.go:414] "containerMap: removing container" podUID="0a27b459-9ad5-4ade-87f3-6ef3ef735e30" containerName="istio-proxy" Oct 31 13:41:51.471204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.471145 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a27b459-9ad5-4ade-87f3-6ef3ef735e30" containerName="istio-proxy" Oct 31 13:41:51.471312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.471303 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a27b459-9ad5-4ade-87f3-6ef3ef735e30" containerName="istio-proxy" Oct 31 13:41:51.476076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.476050 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.486019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.485988 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-35b584-serving-cert\"" Oct 31 13:41:51.486859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.486834 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng"] Oct 31 13:41:51.519867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.519832 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-token\" (UniqueName: \"kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-token\") pod \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " Oct 31 13:41:51.520035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.519895 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-credential-socket\") pod \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " Oct 31 13:41:51.520035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.519927 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-certs\") pod \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " Oct 31 13:41:51.520035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.520007 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-socket\") pod \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " Oct 31 13:41:51.520230 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.520092 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-data\") pod \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " Oct 31 13:41:51.520230 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.520214 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istiod-ca-cert\") pod \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " Oct 31 13:41:51.520365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.520346 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-kube-api-access-v89jw\" (UniqueName: \"kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-kube-api-access-v89jw\") pod \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " Oct 31 13:41:51.520506 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.520490 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-envoy\") pod \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " Oct 31 13:41:51.520573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.520543 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-podinfo\") pod \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\" (UID: \"0a27b459-9ad5-4ade-87f3-6ef3ef735e30\") " Oct 31 13:41:51.520958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.520931 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrkbd\" (UniqueName: \"kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-kube-api-access-mrkbd\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.521043 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521017 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.521108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521088 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.521169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521113 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.521224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521206 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.521309 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521293 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.521366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521344 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.521450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521434 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.521734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521716 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.521831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521817 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.521894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521836 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqrmh\" (UniqueName: \"kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-kube-api-access-dqrmh\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.521894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521853 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.521894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521868 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.521894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521882 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.522118 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.521897 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/226dd9d4-80bc-4c59-8f41-d1cefc341cf9-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.523262 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.523229 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-certs" pod "0a27b459-9ad5-4ade-87f3-6ef3ef735e30" (UID: "0a27b459-9ad5-4ade-87f3-6ef3ef735e30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:51.524032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.524006 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.524142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.524035 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.524472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.524447 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-data" pod "0a27b459-9ad5-4ade-87f3-6ef3ef735e30" (UID: "0a27b459-9ad5-4ade-87f3-6ef3ef735e30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:51.524557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.524501 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istiod-ca-cert" pod "0a27b459-9ad5-4ade-87f3-6ef3ef735e30" (UID: "0a27b459-9ad5-4ade-87f3-6ef3ef735e30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:41:51.524680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.524657 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-socket" pod "0a27b459-9ad5-4ade-87f3-6ef3ef735e30" (UID: "0a27b459-9ad5-4ade-87f3-6ef3ef735e30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:51.524746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.524713 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.524951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.524924 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.525127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.525104 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.525214 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.525194 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.525563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.525536 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-credential-socket" pod "0a27b459-9ad5-4ade-87f3-6ef3ef735e30" (UID: "0a27b459-9ad5-4ade-87f3-6ef3ef735e30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:51.526065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.526023 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-podinfo" pod "0a27b459-9ad5-4ade-87f3-6ef3ef735e30" (UID: "0a27b459-9ad5-4ade-87f3-6ef3ef735e30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:41:51.527748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.527724 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-token" pod "0a27b459-9ad5-4ade-87f3-6ef3ef735e30" (UID: "0a27b459-9ad5-4ade-87f3-6ef3ef735e30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:51.527852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.527809 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.528874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.528852 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-kube-api-access-v89jw" pod "0a27b459-9ad5-4ade-87f3-6ef3ef735e30" (UID: "0a27b459-9ad5-4ade-87f3-6ef3ef735e30"). InnerVolumeSpecName "kube-api-access-v89jw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:41:51.529067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.529052 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-envoy" pod "0a27b459-9ad5-4ade-87f3-6ef3ef735e30" (UID: "0a27b459-9ad5-4ade-87f3-6ef3ef735e30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:41:51.537565 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.537540 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrkbd\" (UniqueName: \"kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-kube-api-access-mrkbd\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.539757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.539732 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.623882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.623849 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8thw6\" (UniqueName: \"kubernetes.io/projected/50a7df40-0977-49e6-9a4a-1669621ee182-kube-api-access-8thw6\") pod \"maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.624057 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.623890 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-maas-api-tls\") pod \"maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.624100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.624058 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-metrics-certs\") pod \"maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.624100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.624081 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a7df40-0977-49e6-9a4a-1669621ee182-openshift-service-ca\") pod \"maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.624412 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.624390 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.624452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.624416 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v89jw\" (UniqueName: \"kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-kube-api-access-v89jw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.624452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.624427 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.624452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.624435 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.624452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.624444 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.624578 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.624453 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.624578 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.624463 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.624578 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.624472 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.624578 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.624482 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a27b459-9ad5-4ade-87f3-6ef3ef735e30-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:41:51.643225 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.643198 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:51.740169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.740138 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-metrics-certs\") pod \"maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.745641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.740498 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a7df40-0977-49e6-9a4a-1669621ee182-openshift-service-ca\") pod \"maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.745641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.741342 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a7df40-0977-49e6-9a4a-1669621ee182-openshift-service-ca\") pod \"maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.745641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.742212 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8thw6\" (UniqueName: \"kubernetes.io/projected/50a7df40-0977-49e6-9a4a-1669621ee182-kube-api-access-8thw6\") pod \"maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.745641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.742573 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-maas-api-tls\") pod \"maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.745641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.745358 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-metrics-certs\") pod \"maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.745987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.745750 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-maas-api-tls\") pod \"maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.762230 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.761913 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8thw6\" (UniqueName: \"kubernetes.io/projected/50a7df40-0977-49e6-9a4a-1669621ee182-kube-api-access-8thw6\") pod \"maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.799470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.799435 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:51.887188 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.887149 5532 generic.go:356] "Generic (PLEG): container finished" podID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" containerID="e71e671ac5d840b7f6ef1b60dfa465bbf836bbfc201d6a62984d1c04851f1017" exitCode=0 Oct 31 13:41:51.887363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.887217 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" Oct 31 13:41:51.887363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.887335 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" event={"ID":"226dd9d4-80bc-4c59-8f41-d1cefc341cf9","Type":"ContainerDied","Data":"e71e671ac5d840b7f6ef1b60dfa465bbf836bbfc201d6a62984d1c04851f1017"} Oct 31 13:41:51.887474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.887404 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc" event={"ID":"226dd9d4-80bc-4c59-8f41-d1cefc341cf9","Type":"ContainerDied","Data":"bbdb756b8338fb1ac99956345f57ea65474c7e32673543d0a8336c165ff30613"} Oct 31 13:41:51.887474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.887430 5532 scope.go:122] "RemoveContainer" containerID="e71e671ac5d840b7f6ef1b60dfa465bbf836bbfc201d6a62984d1c04851f1017" Oct 31 13:41:51.889810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.889783 5532 generic.go:356] "Generic (PLEG): container finished" podID="0a27b459-9ad5-4ade-87f3-6ef3ef735e30" containerID="1ac4a737aaae7c4b67f8dd0ec04ea5c623b623e7db8fbe9f836bfbdd80c25487" exitCode=0 Oct 31 13:41:51.889917 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.889898 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" Oct 31 13:41:51.889994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.889903 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" event={"ID":"0a27b459-9ad5-4ade-87f3-6ef3ef735e30","Type":"ContainerDied","Data":"1ac4a737aaae7c4b67f8dd0ec04ea5c623b623e7db8fbe9f836bfbdd80c25487"} Oct 31 13:41:51.890047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.890030 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t" event={"ID":"0a27b459-9ad5-4ade-87f3-6ef3ef735e30","Type":"ContainerDied","Data":"e6be2a43d63cd1bf93e77bed74229782c6d3a348e2f619c1d371fc02304240f8"} Oct 31 13:41:51.910326 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.910305 5532 scope.go:122] "RemoveContainer" containerID="e71e671ac5d840b7f6ef1b60dfa465bbf836bbfc201d6a62984d1c04851f1017" Oct 31 13:41:51.910714 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:41:51.910686 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e71e671ac5d840b7f6ef1b60dfa465bbf836bbfc201d6a62984d1c04851f1017\": container with ID starting with e71e671ac5d840b7f6ef1b60dfa465bbf836bbfc201d6a62984d1c04851f1017 not found: ID does not exist" containerID="e71e671ac5d840b7f6ef1b60dfa465bbf836bbfc201d6a62984d1c04851f1017" Oct 31 13:41:51.910811 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.910727 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e71e671ac5d840b7f6ef1b60dfa465bbf836bbfc201d6a62984d1c04851f1017"} err="failed to get container status \"e71e671ac5d840b7f6ef1b60dfa465bbf836bbfc201d6a62984d1c04851f1017\": rpc error: code = NotFound desc = could not find container \"e71e671ac5d840b7f6ef1b60dfa465bbf836bbfc201d6a62984d1c04851f1017\": container with ID starting with e71e671ac5d840b7f6ef1b60dfa465bbf836bbfc201d6a62984d1c04851f1017 not found: ID does not exist" Oct 31 13:41:51.910811 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.910748 5532 scope.go:122] "RemoveContainer" containerID="1ac4a737aaae7c4b67f8dd0ec04ea5c623b623e7db8fbe9f836bfbdd80c25487" Oct 31 13:41:51.933858 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.933826 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc"] Oct 31 13:41:51.934016 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.933965 5532 scope.go:122] "RemoveContainer" containerID="1ac4a737aaae7c4b67f8dd0ec04ea5c623b623e7db8fbe9f836bfbdd80c25487" Oct 31 13:41:51.939056 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:41:51.939027 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ac4a737aaae7c4b67f8dd0ec04ea5c623b623e7db8fbe9f836bfbdd80c25487\": container with ID starting with 1ac4a737aaae7c4b67f8dd0ec04ea5c623b623e7db8fbe9f836bfbdd80c25487 not found: ID does not exist" containerID="1ac4a737aaae7c4b67f8dd0ec04ea5c623b623e7db8fbe9f836bfbdd80c25487" Oct 31 13:41:51.939235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.939205 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ac4a737aaae7c4b67f8dd0ec04ea5c623b623e7db8fbe9f836bfbdd80c25487"} err="failed to get container status \"1ac4a737aaae7c4b67f8dd0ec04ea5c623b623e7db8fbe9f836bfbdd80c25487\": rpc error: code = NotFound desc = could not find container \"1ac4a737aaae7c4b67f8dd0ec04ea5c623b623e7db8fbe9f836bfbdd80c25487\": container with ID starting with 1ac4a737aaae7c4b67f8dd0ec04ea5c623b623e7db8fbe9f836bfbdd80c25487 not found: ID does not exist" Oct 31 13:41:51.940506 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.940482 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-vcnzc"] Oct 31 13:41:51.961324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.961297 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff"] Oct 31 13:41:51.961800 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:41:51.961766 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8a86ac3_9fe8_4d9b_b5c9_8669fc03e663.slice/crio-40d92dcdd2414f06c3c9740a234ab7256b2110c7a528999b3e0a29192cd55dce WatchSource:0}: Error finding container 40d92dcdd2414f06c3c9740a234ab7256b2110c7a528999b3e0a29192cd55dce: Status 404 returned error can't find the container with id 40d92dcdd2414f06c3c9740a234ab7256b2110c7a528999b3e0a29192cd55dce Oct 31 13:41:51.966832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.966790 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:41:51.966942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.966880 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:41:51.966942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.966922 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:41:51.967820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.967712 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t"] Oct 31 13:41:51.983996 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:51.983963 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-wrn4t"] Oct 31 13:41:52.014469 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:52.014437 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a27b459-9ad5-4ade-87f3-6ef3ef735e30" path="/var/lib/kubelet/pods/0a27b459-9ad5-4ade-87f3-6ef3ef735e30/volumes" Oct 31 13:41:52.015102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:52.015082 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="226dd9d4-80bc-4c59-8f41-d1cefc341cf9" path="/var/lib/kubelet/pods/226dd9d4-80bc-4c59-8f41-d1cefc341cf9/volumes" Oct 31 13:41:52.159028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:52.158982 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng"] Oct 31 13:41:52.159857 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:41:52.159811 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50a7df40_0977_49e6_9a4a_1669621ee182.slice/crio-4b017d44d477afaf9113041cedad119cca891bff7efb43253e2f02ef2d359568 WatchSource:0}: Error finding container 4b017d44d477afaf9113041cedad119cca891bff7efb43253e2f02ef2d359568: Status 404 returned error can't find the container with id 4b017d44d477afaf9113041cedad119cca891bff7efb43253e2f02ef2d359568 Oct 31 13:41:52.904997 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:52.904961 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" event={"ID":"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663","Type":"ContainerStarted","Data":"447a0378fd40567b30c4e43c7bd10ec38c9f0743b947f30919164e60e01fd5c7"} Oct 31 13:41:52.905492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:52.905017 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" event={"ID":"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663","Type":"ContainerStarted","Data":"40d92dcdd2414f06c3c9740a234ab7256b2110c7a528999b3e0a29192cd55dce"} Oct 31 13:41:52.908194 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:52.908164 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" event={"ID":"50a7df40-0977-49e6-9a4a-1669621ee182","Type":"ContainerStarted","Data":"432e920b01dc2d2e607b123c6aafef3b8bcc23d9654d3d32f3c10e5dbebc47c0"} Oct 31 13:41:52.908329 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:52.908211 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" event={"ID":"50a7df40-0977-49e6-9a4a-1669621ee182","Type":"ContainerStarted","Data":"4b017d44d477afaf9113041cedad119cca891bff7efb43253e2f02ef2d359568"} Oct 31 13:41:52.908329 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:52.908250 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:52.933257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:52.933202 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" podStartSLOduration=146.933187697 podStartE2EDuration="2m26.933187697s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:41:52.931904514 +0000 UTC m=+2281.077648354" watchObservedRunningTime="2026-10-31 13:41:52.933187697 +0000 UTC m=+2281.078931524" Oct 31 13:41:52.956357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:52.956303 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" podStartSLOduration=33.539557916 podStartE2EDuration="33.95628598s" totalImagesPullingTime="416.728064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:41:52.95275182 +0000 UTC m=+2281.098495646" watchObservedRunningTime="2026-10-31 13:41:52.95628598 +0000 UTC m=+2281.102029811" Oct 31 13:41:53.649360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:53.647630 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:53.657973 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:53.657722 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 13:41:53.657973 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:53.657792 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" podUID="b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 13:41:54.647623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:54.644028 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 13:41:54.647623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:54.644094 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" podUID="b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 13:41:55.656774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:55.649955 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 13:41:55.656774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:55.650017 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" podUID="b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 13:41:56.645621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:56.643933 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 13:41:56.645621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:56.643993 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" podUID="b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 13:41:57.659619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:57.658989 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 13:41:57.659619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:57.659053 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" podUID="b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 13:41:58.657109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:58.657064 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" start-of-body= Oct 31 13:41:58.657433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:58.657403 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" podUID="b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.185:15021/healthz/ready\": dial tcp 10.132.0.185:15021: connect: connection refused" Oct 31 13:41:58.925502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:58.925430 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:41:59.650143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:59.650116 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:59.650457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:59.650434 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:41:59.651306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:41:59.651287 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:42:01.763511 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:42:01.763477 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:42:31.763916 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:42:31.763872 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:42:41.268303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:41.268268 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq"] Oct 31 13:42:41.268866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:41.268499 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" podUID="2cb4873b-36ee-48b7-aa65-6966c54d12ec" containerName="istio-proxy" containerID="cri-o://003558e40cf03ea63461e337b426270d71284479b894262eb6533361f4b012fb" gracePeriod=30 Oct 31 13:42:42.059685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:42.059650 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv"] Oct 31 13:42:42.059895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:42.059869 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" podUID="0792431b-9ede-4c06-9d2e-29b8187e13ab" containerName="istio-proxy" containerID="cri-o://d1aac86f225dd13868c44eb0f5eae9d1fc42c532cde8c5a0915a96c393ed973c" gracePeriod=30 Oct 31 13:42:46.048776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.048732 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:42:46.049259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.048815 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" podUID="2cb4873b-36ee-48b7-aa65-6966c54d12ec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:42:46.596980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.596945 5532 generic.go:356] "Generic (PLEG): container finished" podID="2cb4873b-36ee-48b7-aa65-6966c54d12ec" containerID="003558e40cf03ea63461e337b426270d71284479b894262eb6533361f4b012fb" exitCode=0 Oct 31 13:42:46.597295 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.596975 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" event={"ID":"2cb4873b-36ee-48b7-aa65-6966c54d12ec","Type":"ContainerDied","Data":"003558e40cf03ea63461e337b426270d71284479b894262eb6533361f4b012fb"} Oct 31 13:42:46.631617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.631577 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:42:46.672570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.672538 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx"] Oct 31 13:42:46.674063 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.674038 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2cb4873b-36ee-48b7-aa65-6966c54d12ec" containerName="istio-proxy" Oct 31 13:42:46.674063 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.674067 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cb4873b-36ee-48b7-aa65-6966c54d12ec" containerName="istio-proxy" Oct 31 13:42:46.674355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.674336 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cb4873b-36ee-48b7-aa65-6966c54d12ec" containerName="istio-proxy" Oct 31 13:42:46.679258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.679234 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.692582 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.692553 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx"] Oct 31 13:42:46.727773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.727736 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-data\") pod \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " Oct 31 13:42:46.727945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.727787 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-socket\") pod \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " Oct 31 13:42:46.727945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.727881 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-envoy\") pod \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " Oct 31 13:42:46.728086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.727945 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-kube-api-access-v8k8f\" (UniqueName: \"kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-kube-api-access-v8k8f\") pod \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " Oct 31 13:42:46.728152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.728106 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-socket" pod "2cb4873b-36ee-48b7-aa65-6966c54d12ec" (UID: "2cb4873b-36ee-48b7-aa65-6966c54d12ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:46.728152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.728059 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-data" pod "2cb4873b-36ee-48b7-aa65-6966c54d12ec" (UID: "2cb4873b-36ee-48b7-aa65-6966c54d12ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:46.728271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.728172 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istiod-ca-cert\") pod \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " Oct 31 13:42:46.728271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.728221 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-certs\") pod \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " Oct 31 13:42:46.728271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.728261 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-token\" (UniqueName: \"kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-token\") pod \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " Oct 31 13:42:46.728432 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.728307 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-podinfo\") pod \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " Oct 31 13:42:46.728432 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.728383 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-credential-socket\") pod \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\" (UID: \"2cb4873b-36ee-48b7-aa65-6966c54d12ec\") " Oct 31 13:42:46.729514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.728575 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-certs" pod "2cb4873b-36ee-48b7-aa65-6966c54d12ec" (UID: "2cb4873b-36ee-48b7-aa65-6966c54d12ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:46.729514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.729273 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-credential-socket" pod "2cb4873b-36ee-48b7-aa65-6966c54d12ec" (UID: "2cb4873b-36ee-48b7-aa65-6966c54d12ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:46.729514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.729372 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istiod-ca-cert" pod "2cb4873b-36ee-48b7-aa65-6966c54d12ec" (UID: "2cb4873b-36ee-48b7-aa65-6966c54d12ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:46.729514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.729389 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:46.729514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.729403 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:46.729514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.729412 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:46.729514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.729421 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:46.731451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.731425 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-token" pod "2cb4873b-36ee-48b7-aa65-6966c54d12ec" (UID: "2cb4873b-36ee-48b7-aa65-6966c54d12ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:46.731560 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.731463 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-podinfo" pod "2cb4873b-36ee-48b7-aa65-6966c54d12ec" (UID: "2cb4873b-36ee-48b7-aa65-6966c54d12ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:42:46.732982 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.732958 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-kube-api-access-v8k8f" pod "2cb4873b-36ee-48b7-aa65-6966c54d12ec" (UID: "2cb4873b-36ee-48b7-aa65-6966c54d12ec"). InnerVolumeSpecName "kube-api-access-v8k8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:46.734531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.734505 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-envoy" pod "2cb4873b-36ee-48b7-aa65-6966c54d12ec" (UID: "2cb4873b-36ee-48b7-aa65-6966c54d12ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:46.831296 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.831262 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.831477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.831393 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.831535 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.831520 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.831573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.831558 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.831723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.831706 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe5d1c7-179a-42e0-8457-3cba517be508-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.831778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.831764 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.831857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.831843 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.831940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.831926 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.831989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.831957 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5k22\" (UniqueName: \"kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-kube-api-access-x5k22\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.832112 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.832099 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:46.832149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.832116 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8k8f\" (UniqueName: \"kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-kube-api-access-v8k8f\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:46.832149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.832127 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:46.832149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.832137 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:46.832149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.832145 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cb4873b-36ee-48b7-aa65-6966c54d12ec-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:46.933579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.933540 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe5d1c7-179a-42e0-8457-3cba517be508-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.933757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.933646 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.933757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.933703 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.933908 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.933777 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.933908 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.933804 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5k22\" (UniqueName: \"kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-kube-api-access-x5k22\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.934021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.933921 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.934093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.934027 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.934155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.934109 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.934212 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.934179 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.934270 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.934219 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.934270 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.934175 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.934443 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.934393 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe5d1c7-179a-42e0-8457-3cba517be508-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.934513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.934453 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.934669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.934647 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.936263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.936239 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.936674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.936644 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.947065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.947040 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:46.947161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:46.947149 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5k22\" (UniqueName: \"kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-kube-api-access-x5k22\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:47.013398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.013375 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:47.365104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.365062 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx"] Oct 31 13:42:47.367153 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:42:47.367115 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe5d1c7_179a_42e0_8457_3cba517be508.slice/crio-dbef5d3d5368f153d772a42e68f6ad32b15eb99e2ed76f0acf560caed692ee2c WatchSource:0}: Error finding container dbef5d3d5368f153d772a42e68f6ad32b15eb99e2ed76f0acf560caed692ee2c: Status 404 returned error can't find the container with id dbef5d3d5368f153d772a42e68f6ad32b15eb99e2ed76f0acf560caed692ee2c Oct 31 13:42:47.374059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.373969 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:42:47.374644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.374616 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:42:47.374826 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.374806 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:42:47.523524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.523493 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:42:47.583408 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.581252 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl"] Oct 31 13:42:47.583408 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.582792 5532 cpu_manager.go:414] "containerMap: removing container" podUID="0792431b-9ede-4c06-9d2e-29b8187e13ab" containerName="istio-proxy" Oct 31 13:42:47.583408 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.582814 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0792431b-9ede-4c06-9d2e-29b8187e13ab" containerName="istio-proxy" Oct 31 13:42:47.583408 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.583135 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="0792431b-9ede-4c06-9d2e-29b8187e13ab" containerName="istio-proxy" Oct 31 13:42:47.588303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.588276 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.627120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.627086 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl"] Oct 31 13:42:47.636127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.636099 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" event={"ID":"bfe5d1c7-179a-42e0-8457-3cba517be508","Type":"ContainerStarted","Data":"5ee26139b61dc69334a980b4c383da3c7af64013ccb75bbe61c57406599e927a"} Oct 31 13:42:47.636321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.636308 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" event={"ID":"bfe5d1c7-179a-42e0-8457-3cba517be508","Type":"ContainerStarted","Data":"dbef5d3d5368f153d772a42e68f6ad32b15eb99e2ed76f0acf560caed692ee2c"} Oct 31 13:42:47.639392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.639366 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" event={"ID":"2cb4873b-36ee-48b7-aa65-6966c54d12ec","Type":"ContainerDied","Data":"91080f49d1e9bab52a230e2eaef0d2880dd189f82a6f35af5c849be250041e25"} Oct 31 13:42:47.640048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.640033 5532 scope.go:122] "RemoveContainer" containerID="003558e40cf03ea63461e337b426270d71284479b894262eb6533361f4b012fb" Oct 31 13:42:47.640617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.639536 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq" Oct 31 13:42:47.642373 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.642314 5532 generic.go:356] "Generic (PLEG): container finished" podID="0792431b-9ede-4c06-9d2e-29b8187e13ab" containerID="d1aac86f225dd13868c44eb0f5eae9d1fc42c532cde8c5a0915a96c393ed973c" exitCode=0 Oct 31 13:42:47.642524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.642504 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" event={"ID":"0792431b-9ede-4c06-9d2e-29b8187e13ab","Type":"ContainerDied","Data":"d1aac86f225dd13868c44eb0f5eae9d1fc42c532cde8c5a0915a96c393ed973c"} Oct 31 13:42:47.642632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.642551 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" event={"ID":"0792431b-9ede-4c06-9d2e-29b8187e13ab","Type":"ContainerDied","Data":"d0027832dfc345e6d01134d4953c4c6a250fc262e8ec4f18f417baed329932e5"} Oct 31 13:42:47.642704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.642670 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv" Oct 31 13:42:47.658230 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.658207 5532 scope.go:122] "RemoveContainer" containerID="d1aac86f225dd13868c44eb0f5eae9d1fc42c532cde8c5a0915a96c393ed973c" Oct 31 13:42:47.661466 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.661438 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-credential-socket\") pod \"0792431b-9ede-4c06-9d2e-29b8187e13ab\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " Oct 31 13:42:47.661569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.661510 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-certs\") pod \"0792431b-9ede-4c06-9d2e-29b8187e13ab\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " Oct 31 13:42:47.661682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.661649 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0792431b-9ede-4c06-9d2e-29b8187e13ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0792431b-9ede-4c06-9d2e-29b8187e13ab-istiod-ca-cert\") pod \"0792431b-9ede-4c06-9d2e-29b8187e13ab\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " Oct 31 13:42:47.661774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.661760 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-data\") pod \"0792431b-9ede-4c06-9d2e-29b8187e13ab\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " Oct 31 13:42:47.661835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.661812 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-podinfo\") pod \"0792431b-9ede-4c06-9d2e-29b8187e13ab\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " Oct 31 13:42:47.661889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.661846 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-socket\") pod \"0792431b-9ede-4c06-9d2e-29b8187e13ab\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " Oct 31 13:42:47.661948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.661922 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-kube-api-access-kbzw2\" (UniqueName: \"kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-kube-api-access-kbzw2\") pod \"0792431b-9ede-4c06-9d2e-29b8187e13ab\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " Oct 31 13:42:47.662112 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.662097 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-token\" (UniqueName: \"kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-token\") pod \"0792431b-9ede-4c06-9d2e-29b8187e13ab\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " Oct 31 13:42:47.662205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.662192 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-envoy\") pod \"0792431b-9ede-4c06-9d2e-29b8187e13ab\" (UID: \"0792431b-9ede-4c06-9d2e-29b8187e13ab\") " Oct 31 13:42:47.662550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.662528 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e147e116-ed36-4be3-82ee-b37c85f31665-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.664014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.662565 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.664014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.662685 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.664014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.662733 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.664014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.662767 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.664014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.662843 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e147e116-ed36-4be3-82ee-b37c85f31665-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.664014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.662871 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbm8h\" (UniqueName: \"kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-kube-api-access-rbm8h\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.664014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.662995 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.686679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.665251 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-socket" pod "0792431b-9ede-4c06-9d2e-29b8187e13ab" (UID: "0792431b-9ede-4c06-9d2e-29b8187e13ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:47.686679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.667754 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-certs" pod "0792431b-9ede-4c06-9d2e-29b8187e13ab" (UID: "0792431b-9ede-4c06-9d2e-29b8187e13ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:47.686679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.667963 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-credential-socket" pod "0792431b-9ede-4c06-9d2e-29b8187e13ab" (UID: "0792431b-9ede-4c06-9d2e-29b8187e13ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:47.686679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.669633 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0792431b-9ede-4c06-9d2e-29b8187e13ab-istiod-ca-cert" pod "0792431b-9ede-4c06-9d2e-29b8187e13ab" (UID: "0792431b-9ede-4c06-9d2e-29b8187e13ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:42:47.686679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.669792 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-data" pod "0792431b-9ede-4c06-9d2e-29b8187e13ab" (UID: "0792431b-9ede-4c06-9d2e-29b8187e13ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:47.686679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.671877 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.686679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.674897 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:47.686679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.674954 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:47.686679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.674972 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0792431b-9ede-4c06-9d2e-29b8187e13ab-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:47.686679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.674988 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:47.686679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.675002 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:47.694589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.694539 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" podStartSLOduration=201.694520421 podStartE2EDuration="3m21.694520421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:47.670340179 +0000 UTC m=+2335.816084007" watchObservedRunningTime="2026-10-31 13:42:47.694520421 +0000 UTC m=+2335.840264250" Oct 31 13:42:47.695267 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.695086 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-token" pod "0792431b-9ede-4c06-9d2e-29b8187e13ab" (UID: "0792431b-9ede-4c06-9d2e-29b8187e13ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:47.701854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.698732 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-podinfo" pod "0792431b-9ede-4c06-9d2e-29b8187e13ab" (UID: "0792431b-9ede-4c06-9d2e-29b8187e13ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:42:47.701854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.700232 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-envoy" pod "0792431b-9ede-4c06-9d2e-29b8187e13ab" (UID: "0792431b-9ede-4c06-9d2e-29b8187e13ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:42:47.711762 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.711733 5532 scope.go:122] "RemoveContainer" containerID="d1aac86f225dd13868c44eb0f5eae9d1fc42c532cde8c5a0915a96c393ed973c" Oct 31 13:42:47.711884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.711730 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-kube-api-access-kbzw2" pod "0792431b-9ede-4c06-9d2e-29b8187e13ab" (UID: "0792431b-9ede-4c06-9d2e-29b8187e13ab"). InnerVolumeSpecName "kube-api-access-kbzw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:42:47.712568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.712542 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq"] Oct 31 13:42:47.715341 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:42:47.715311 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1aac86f225dd13868c44eb0f5eae9d1fc42c532cde8c5a0915a96c393ed973c\": container with ID starting with d1aac86f225dd13868c44eb0f5eae9d1fc42c532cde8c5a0915a96c393ed973c not found: ID does not exist" containerID="d1aac86f225dd13868c44eb0f5eae9d1fc42c532cde8c5a0915a96c393ed973c" Oct 31 13:42:47.715505 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.715477 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1aac86f225dd13868c44eb0f5eae9d1fc42c532cde8c5a0915a96c393ed973c"} err="failed to get container status \"d1aac86f225dd13868c44eb0f5eae9d1fc42c532cde8c5a0915a96c393ed973c\": rpc error: code = NotFound desc = could not find container \"d1aac86f225dd13868c44eb0f5eae9d1fc42c532cde8c5a0915a96c393ed973c\": container with ID starting with d1aac86f225dd13868c44eb0f5eae9d1fc42c532cde8c5a0915a96c393ed973c not found: ID does not exist" Oct 31 13:42:47.718975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.718952 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-vl7kq"] Oct 31 13:42:47.777139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.777047 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.777306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.777247 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e147e116-ed36-4be3-82ee-b37c85f31665-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.777386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.777362 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.777522 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.777498 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.777607 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.777558 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.777680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.777639 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.777741 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.777732 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e147e116-ed36-4be3-82ee-b37c85f31665-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.777812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.777761 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbm8h\" (UniqueName: \"kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-kube-api-access-rbm8h\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.777943 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.777919 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.778078 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.778054 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e147e116-ed36-4be3-82ee-b37c85f31665-istiod-ca-cert\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.778294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.778252 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-data\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.778388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.778364 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-credential-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.778442 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.778419 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-socket\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.778487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.778455 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:47.778529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.778485 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:47.778529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.778502 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbzw2\" (UniqueName: \"kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-kube-api-access-kbzw2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:47.778529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.778517 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0792431b-9ede-4c06-9d2e-29b8187e13ab-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:42:47.778841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.778821 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-certs\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.780436 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.780409 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-envoy\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.780578 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.780558 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e147e116-ed36-4be3-82ee-b37c85f31665-istio-podinfo\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.793182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.793159 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-istio-token\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.793320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.793217 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbm8h\" (UniqueName: \"kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-kube-api-access-rbm8h\") pod \"e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.926840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.926810 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:47.978246 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.978214 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv"] Oct 31 13:42:47.987476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:47.987419 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-sclgv"] Oct 31 13:42:48.014855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:48.013353 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0792431b-9ede-4c06-9d2e-29b8187e13ab" path="/var/lib/kubelet/pods/0792431b-9ede-4c06-9d2e-29b8187e13ab/volumes" Oct 31 13:42:48.014855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:48.014254 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cb4873b-36ee-48b7-aa65-6966c54d12ec" path="/var/lib/kubelet/pods/2cb4873b-36ee-48b7-aa65-6966c54d12ec/volumes" Oct 31 13:42:48.014855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:48.014798 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:48.015498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:48.015472 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:42:48.015983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:48.015523 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 31 13:42:48.285778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:48.285711 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl"] Oct 31 13:42:48.288714 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:42:48.288682 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode147e116_ed36_4be3_82ee_b37c85f31665.slice/crio-d4af76cf47f667235ee1272f2309605e6ff1afa553aef228d7e56387c19bae47 WatchSource:0}: Error finding container d4af76cf47f667235ee1272f2309605e6ff1afa553aef228d7e56387c19bae47: Status 404 returned error can't find the container with id d4af76cf47f667235ee1272f2309605e6ff1afa553aef228d7e56387c19bae47 Oct 31 13:42:48.298407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:48.294733 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:42:48.298407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:48.294827 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:42:48.298407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:48.294876 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:42:48.674808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:48.674761 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" event={"ID":"e147e116-ed36-4be3-82ee-b37c85f31665","Type":"ContainerStarted","Data":"33cf7bb242187c5c0d325c65385fb019605feb924f1271e15e090eca3c6b17a8"} Oct 31 13:42:48.675263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:48.674838 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" event={"ID":"e147e116-ed36-4be3-82ee-b37c85f31665","Type":"ContainerStarted","Data":"d4af76cf47f667235ee1272f2309605e6ff1afa553aef228d7e56387c19bae47"} Oct 31 13:42:48.706157 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:48.706111 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" podStartSLOduration=187.706096552 podStartE2EDuration="3m7.706096552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:42:48.70064496 +0000 UTC m=+2336.846388789" watchObservedRunningTime="2026-10-31 13:42:48.706096552 +0000 UTC m=+2336.851840451" Oct 31 13:42:48.927268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:48.927192 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:49.014701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:49.014671 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:42:49.014870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:49.014720 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 31 13:42:49.928870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:49.928838 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:42:49.929307 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:49.928904 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" podUID="e147e116-ed36-4be3-82ee-b37c85f31665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 13:42:50.016084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:50.016048 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:42:50.016348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:50.016320 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 31 13:42:50.928740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:50.928083 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:42:50.928740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:50.928148 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" podUID="e147e116-ed36-4be3-82ee-b37c85f31665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 13:42:51.012496 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:51.012462 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:42:51.014151 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:51.014126 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:42:51.014209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:51.014176 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 31 13:42:51.927487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:51.927449 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:42:51.927766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:51.927508 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" podUID="e147e116-ed36-4be3-82ee-b37c85f31665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 13:42:52.014386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:52.014351 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:42:52.014826 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:52.014400 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 31 13:42:52.930616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:52.930326 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:42:52.930616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:52.930387 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" podUID="e147e116-ed36-4be3-82ee-b37c85f31665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 13:42:53.026617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:53.026324 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:42:53.026617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:53.026385 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 31 13:42:53.321852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:53.321761 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:42:53.323002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:53.322970 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" podUID="fb8e0c0b-e52a-4752-93e9-2624bbde361b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.178:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:42:53.760734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:53.760690 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:42:53.936617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:53.929990 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:42:53.936617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:53.930053 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" podUID="e147e116-ed36-4be3-82ee-b37c85f31665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 13:42:54.049828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.049731 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:42:54.148950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.148905 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 31 13:42:54.343435 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.343390 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" podUID="fb8e0c0b-e52a-4752-93e9-2624bbde361b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.178:8000/ready\": context deadline exceeded" Oct 31 13:42:54.373916 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.373880 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:42:54.404724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.404676 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:42:54.405445 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.405408 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:42:54.423621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.416669 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" podUID="8acd1643-d958-467c-8055-de8d1fff7ef2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.179:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:42:54.423621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.417382 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:42:54.450209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.450163 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:42:54.516414 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.516370 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": context deadline exceeded" Oct 31 13:42:54.518329 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.518292 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:42:54.550202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.549405 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:42:54.679378 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.679341 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:42:54.725802 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.725748 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:42:54.778240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.778190 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:42:54.859621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:54.779689 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:42:55.159681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.151776 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/health\": context deadline exceeded" Oct 31 13:42:55.199629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.199169 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 13:42:55.233624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.210217 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:42:55.233624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.210298 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 31 13:42:55.334621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.301344 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:42:55.334621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.301409 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" podUID="e147e116-ed36-4be3-82ee-b37c85f31665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 13:42:55.373649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.373554 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:42:55.388394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.386788 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 13:42:55.457425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.457309 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:42:55.459445 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.459420 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:42:55.461769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.461728 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:42:55.593656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.589233 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:42:55.593656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.589969 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:42:55.593656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.590034 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:42:55.614568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.614525 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:42:55.937504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.937463 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" start-of-body= Oct 31 13:42:55.937756 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:55.937521 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" podUID="e147e116-ed36-4be3-82ee-b37c85f31665" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.188:15021/healthz/ready\": dial tcp 10.132.0.188:15021: connect: connection refused" Oct 31 13:42:56.217001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:56.216909 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:56.218166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:56.218142 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:56.218849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:56.218828 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:42:57.478122 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:57.478084 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:57.478684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:57.478573 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:42:57.479472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:42:57.479446 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:43:01.765874 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:43:01.765799 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:43:31.887174 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:43:31.887133 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:43:54.435217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:43:54.435168 5532 scope.go:122] "RemoveContainer" containerID="1a12ca69ccc8db0bbb70491c3f93d13cc9d8a8d0103c477d56687a8b2cd8509c" Oct 31 13:44:00.773298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:00.773264 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5"] Oct 31 13:44:00.773759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:00.773486 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" containerID="cri-o://cb70eabf0f4a3973b1e09e720ad54412ce4f38b87d9d8fb3770624996a7cdb49" gracePeriod=30 Oct 31 13:44:00.829050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:00.829018 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d"] Oct 31 13:44:00.829256 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:00.829230 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerName="istio-proxy" containerID="cri-o://7bde2f98e7539ec838be01a01dfb79dbe886ddd83aca39c3fe45cfa77aedfdd0" gracePeriod=30 Oct 31 13:44:01.021449 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:01.021417 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j"] Oct 31 13:44:01.021640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:01.021616 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" podUID="92a204c6-cc2c-40ff-8c62-e5648d4280e1" containerName="istio-proxy" containerID="cri-o://0f1abfd9ff5a19595d96eb79954e4d0a54033163e37e4f6c69353569ba03cc61" gracePeriod=30 Oct 31 13:44:01.764141 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:44:01.764097 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:44:06.164170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.163762 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:44:06.224728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.224695 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22"] Oct 31 13:44:06.225763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.225737 5532 cpu_manager.go:414] "containerMap: removing container" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" Oct 31 13:44:06.225763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.225761 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" Oct 31 13:44:06.226066 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.226048 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerName="istio-proxy" Oct 31 13:44:06.227065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.226957 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:44:06.230435 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.230418 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.241960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.241509 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-socket\") pod \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " Oct 31 13:44:06.241960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.241616 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-podinfo\") pod \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " Oct 31 13:44:06.241960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.241911 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-socket" pod "7bb8d480-1daf-41eb-aab7-e988dc35c2d6" (UID: "7bb8d480-1daf-41eb-aab7-e988dc35c2d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.242394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.242373 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-envoy\") pod \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " Oct 31 13:44:06.242494 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.242448 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istiod-ca-cert\") pod \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " Oct 31 13:44:06.242494 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.242481 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-token\" (UniqueName: \"kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-token\") pod \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " Oct 31 13:44:06.242658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.242562 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-data\") pod \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " Oct 31 13:44:06.242658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.242633 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-credential-socket\") pod \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " Oct 31 13:44:06.242772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.242736 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-kube-api-access-lmh2r\" (UniqueName: \"kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-kube-api-access-lmh2r\") pod \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " Oct 31 13:44:06.242828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.242812 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-certs\") pod \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\" (UID: \"7bb8d480-1daf-41eb-aab7-e988dc35c2d6\") " Oct 31 13:44:06.244173 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.244147 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istiod-ca-cert" pod "7bb8d480-1daf-41eb-aab7-e988dc35c2d6" (UID: "7bb8d480-1daf-41eb-aab7-e988dc35c2d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:06.244497 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.244364 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-credential-socket" pod "7bb8d480-1daf-41eb-aab7-e988dc35c2d6" (UID: "7bb8d480-1daf-41eb-aab7-e988dc35c2d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.244909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.244647 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-data" pod "7bb8d480-1daf-41eb-aab7-e988dc35c2d6" (UID: "7bb8d480-1daf-41eb-aab7-e988dc35c2d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.246642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.245385 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-certs" pod "7bb8d480-1daf-41eb-aab7-e988dc35c2d6" (UID: "7bb8d480-1daf-41eb-aab7-e988dc35c2d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.246642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.245795 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.246642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.245818 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.246642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.245831 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.246642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.245842 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.246642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.245854 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.251005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.250985 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-69bdf5-openshift-default-dockercfg-v2qtd\"" Oct 31 13:44:06.255517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.255489 5532 generic.go:356] "Generic (PLEG): container finished" podID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" containerID="cb70eabf0f4a3973b1e09e720ad54412ce4f38b87d9d8fb3770624996a7cdb49" exitCode=0 Oct 31 13:44:06.255678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.255552 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" Oct 31 13:44:06.255991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.255776 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" event={"ID":"7bb8d480-1daf-41eb-aab7-e988dc35c2d6","Type":"ContainerDied","Data":"cb70eabf0f4a3973b1e09e720ad54412ce4f38b87d9d8fb3770624996a7cdb49"} Oct 31 13:44:06.255991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.255838 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5" event={"ID":"7bb8d480-1daf-41eb-aab7-e988dc35c2d6","Type":"ContainerDied","Data":"76a5752d72c5809913a266b2a25c5cbecc7424ca96e581462015d3383d18f406"} Oct 31 13:44:06.255991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.255866 5532 scope.go:122] "RemoveContainer" containerID="cb70eabf0f4a3973b1e09e720ad54412ce4f38b87d9d8fb3770624996a7cdb49" Oct 31 13:44:06.259985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.259965 5532 generic.go:356] "Generic (PLEG): container finished" podID="92a204c6-cc2c-40ff-8c62-e5648d4280e1" containerID="0f1abfd9ff5a19595d96eb79954e4d0a54033163e37e4f6c69353569ba03cc61" exitCode=0 Oct 31 13:44:06.260241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.260112 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" event={"ID":"92a204c6-cc2c-40ff-8c62-e5648d4280e1","Type":"ContainerDied","Data":"0f1abfd9ff5a19595d96eb79954e4d0a54033163e37e4f6c69353569ba03cc61"} Oct 31 13:44:06.262920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.262442 5532 generic.go:356] "Generic (PLEG): container finished" podID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerID="7bde2f98e7539ec838be01a01dfb79dbe886ddd83aca39c3fe45cfa77aedfdd0" exitCode=0 Oct 31 13:44:06.262920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.262543 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" event={"ID":"2b1ba1d2-444c-4dfc-adea-6458e9e4c590","Type":"ContainerDied","Data":"7bde2f98e7539ec838be01a01dfb79dbe886ddd83aca39c3fe45cfa77aedfdd0"} Oct 31 13:44:06.262920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.262580 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" event={"ID":"2b1ba1d2-444c-4dfc-adea-6458e9e4c590","Type":"ContainerDied","Data":"3a193ac8f5ee80c75b89b62b5fda48ce21d0667bc0c14374c20d431fc9551a3e"} Oct 31 13:44:06.262920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.262673 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d" Oct 31 13:44:06.262920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.262887 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-envoy" pod "7bb8d480-1daf-41eb-aab7-e988dc35c2d6" (UID: "7bb8d480-1daf-41eb-aab7-e988dc35c2d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.263403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.263367 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-kube-api-access-lmh2r" pod "7bb8d480-1daf-41eb-aab7-e988dc35c2d6" (UID: "7bb8d480-1daf-41eb-aab7-e988dc35c2d6"). InnerVolumeSpecName "kube-api-access-lmh2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:06.267524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.267432 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22"] Oct 31 13:44:06.270265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.270242 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-podinfo" pod "7bb8d480-1daf-41eb-aab7-e988dc35c2d6" (UID: "7bb8d480-1daf-41eb-aab7-e988dc35c2d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:06.270683 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.270662 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-token" pod "7bb8d480-1daf-41eb-aab7-e988dc35c2d6" (UID: "7bb8d480-1daf-41eb-aab7-e988dc35c2d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:06.272648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.272589 5532 scope.go:122] "RemoveContainer" containerID="cb70eabf0f4a3973b1e09e720ad54412ce4f38b87d9d8fb3770624996a7cdb49" Oct 31 13:44:06.273041 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:44:06.272962 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb70eabf0f4a3973b1e09e720ad54412ce4f38b87d9d8fb3770624996a7cdb49\": container with ID starting with cb70eabf0f4a3973b1e09e720ad54412ce4f38b87d9d8fb3770624996a7cdb49 not found: ID does not exist" containerID="cb70eabf0f4a3973b1e09e720ad54412ce4f38b87d9d8fb3770624996a7cdb49" Oct 31 13:44:06.273041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.272995 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb70eabf0f4a3973b1e09e720ad54412ce4f38b87d9d8fb3770624996a7cdb49"} err="failed to get container status \"cb70eabf0f4a3973b1e09e720ad54412ce4f38b87d9d8fb3770624996a7cdb49\": rpc error: code = NotFound desc = could not find container \"cb70eabf0f4a3973b1e09e720ad54412ce4f38b87d9d8fb3770624996a7cdb49\": container with ID starting with cb70eabf0f4a3973b1e09e720ad54412ce4f38b87d9d8fb3770624996a7cdb49 not found: ID does not exist" Oct 31 13:44:06.273041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.273018 5532 scope.go:122] "RemoveContainer" containerID="7bde2f98e7539ec838be01a01dfb79dbe886ddd83aca39c3fe45cfa77aedfdd0" Oct 31 13:44:06.288287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.288211 5532 scope.go:122] "RemoveContainer" containerID="7bde2f98e7539ec838be01a01dfb79dbe886ddd83aca39c3fe45cfa77aedfdd0" Oct 31 13:44:06.288579 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:44:06.288560 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bde2f98e7539ec838be01a01dfb79dbe886ddd83aca39c3fe45cfa77aedfdd0\": container with ID starting with 7bde2f98e7539ec838be01a01dfb79dbe886ddd83aca39c3fe45cfa77aedfdd0 not found: ID does not exist" containerID="7bde2f98e7539ec838be01a01dfb79dbe886ddd83aca39c3fe45cfa77aedfdd0" Oct 31 13:44:06.288718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.288589 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bde2f98e7539ec838be01a01dfb79dbe886ddd83aca39c3fe45cfa77aedfdd0"} err="failed to get container status \"7bde2f98e7539ec838be01a01dfb79dbe886ddd83aca39c3fe45cfa77aedfdd0\": rpc error: code = NotFound desc = could not find container \"7bde2f98e7539ec838be01a01dfb79dbe886ddd83aca39c3fe45cfa77aedfdd0\": container with ID starting with 7bde2f98e7539ec838be01a01dfb79dbe886ddd83aca39c3fe45cfa77aedfdd0 not found: ID does not exist" Oct 31 13:44:06.301895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.301825 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz"] Oct 31 13:44:06.303678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.303657 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerName="istio-proxy" Oct 31 13:44:06.303678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.303677 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerName="istio-proxy" Oct 31 13:44:06.306958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.306937 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" containerName="istio-proxy" Oct 31 13:44:06.314293 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.314268 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.328895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.328865 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz"] Oct 31 13:44:06.340844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.340806 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-72c210-openshift-default-dockercfg-bnm6m\"" Oct 31 13:44:06.347154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.347123 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-podinfo\") pod \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " Oct 31 13:44:06.347338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.347247 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istiod-ca-cert\") pod \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " Oct 31 13:44:06.347338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.347287 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-credential-socket\") pod \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " Oct 31 13:44:06.347338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.347329 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-certs\") pod \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " Oct 31 13:44:06.347558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.347486 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-token\" (UniqueName: \"kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-token\") pod \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " Oct 31 13:44:06.347663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.347642 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-kube-api-access-w225v\" (UniqueName: \"kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-kube-api-access-w225v\") pod \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " Oct 31 13:44:06.347768 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.347747 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-credential-socket" pod "2b1ba1d2-444c-4dfc-adea-6458e9e4c590" (UID: "2b1ba1d2-444c-4dfc-adea-6458e9e4c590"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.347934 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.347915 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-envoy\") pod \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " Oct 31 13:44:06.348008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.347950 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-socket\") pod \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " Oct 31 13:44:06.348064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.348031 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-data\") pod \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\" (UID: \"2b1ba1d2-444c-4dfc-adea-6458e9e4c590\") " Oct 31 13:44:06.348402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.348374 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-credential-socket\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.348643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.348621 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-credential-socket\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.348867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.347916 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-certs" pod "2b1ba1d2-444c-4dfc-adea-6458e9e4c590" (UID: "2b1ba1d2-444c-4dfc-adea-6458e9e4c590"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.348867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.348761 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-socket" pod "2b1ba1d2-444c-4dfc-adea-6458e9e4c590" (UID: "2b1ba1d2-444c-4dfc-adea-6458e9e4c590"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.348867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.348793 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-certs\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.348867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.348838 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-data" pod "2b1ba1d2-444c-4dfc-adea-6458e9e4c590" (UID: "2b1ba1d2-444c-4dfc-adea-6458e9e4c590"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.349114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.348899 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zskqs\" (UniqueName: \"kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-kube-api-access-zskqs\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.349114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.348969 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istiod-ca-cert\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.349114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349045 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-data\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.349114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349110 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-envoy\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.349740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349138 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-token\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.349740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349198 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-socket\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.349740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349332 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-socket\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.349740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349372 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-certs\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.349740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349405 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw5l2\" (UniqueName: \"kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-kube-api-access-mw5l2\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.349740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349437 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-data\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.349740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349474 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60eb6552-2f31-4bd4-bfd7-9286e7620151-istiod-ca-cert\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.349740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349536 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-token\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.349740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349637 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-envoy\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.349740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349727 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-podinfo\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.350257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349805 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-podinfo\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.350257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349964 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmh2r\" (UniqueName: \"kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-kube-api-access-lmh2r\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.350257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349984 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.350257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.349999 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.350257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.350014 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.350257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.350029 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.350257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.350043 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.350257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.350056 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.350257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.350070 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bb8d480-1daf-41eb-aab7-e988dc35c2d6-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.351586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.351561 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istiod-ca-cert" pod "2b1ba1d2-444c-4dfc-adea-6458e9e4c590" (UID: "2b1ba1d2-444c-4dfc-adea-6458e9e4c590"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:06.352936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.352874 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-token" pod "2b1ba1d2-444c-4dfc-adea-6458e9e4c590" (UID: "2b1ba1d2-444c-4dfc-adea-6458e9e4c590"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:06.352936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.352921 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-kube-api-access-w225v" pod "2b1ba1d2-444c-4dfc-adea-6458e9e4c590" (UID: "2b1ba1d2-444c-4dfc-adea-6458e9e4c590"). InnerVolumeSpecName "kube-api-access-w225v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:06.355153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.355129 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-envoy" pod "2b1ba1d2-444c-4dfc-adea-6458e9e4c590" (UID: "2b1ba1d2-444c-4dfc-adea-6458e9e4c590"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.355788 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.355767 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-podinfo" pod "2b1ba1d2-444c-4dfc-adea-6458e9e4c590" (UID: "2b1ba1d2-444c-4dfc-adea-6458e9e4c590"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:06.413119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.413086 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:44:06.451398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.451321 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-socket\") pod \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " Oct 31 13:44:06.451985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.451960 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-token\" (UniqueName: \"kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-token\") pod \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " Oct 31 13:44:06.452608 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.451886 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-socket" pod "92a204c6-cc2c-40ff-8c62-e5648d4280e1" (UID: "92a204c6-cc2c-40ff-8c62-e5648d4280e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.452784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.452756 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istiod-ca-cert\") pod \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.452830 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-podinfo\") pod \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.452910 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-kube-api-access-qxnl8\" (UniqueName: \"kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-kube-api-access-qxnl8\") pod \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.453097 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-credential-socket\") pod \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.453117 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istiod-ca-cert" pod "92a204c6-cc2c-40ff-8c62-e5648d4280e1" (UID: "92a204c6-cc2c-40ff-8c62-e5648d4280e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.453155 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-envoy\") pod \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.453205 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-certs\") pod \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.453437 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-data\") pod \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\" (UID: \"92a204c6-cc2c-40ff-8c62-e5648d4280e1\") " Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.453766 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-data\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.453843 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-envoy\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.453872 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-token\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.453919 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-socket\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.453997 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-credential-socket" pod "92a204c6-cc2c-40ff-8c62-e5648d4280e1" (UID: "92a204c6-cc2c-40ff-8c62-e5648d4280e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.454073 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-socket\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.454115 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-certs\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.454149 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw5l2\" (UniqueName: \"kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-kube-api-access-mw5l2\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.454187 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-data\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.454231 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60eb6552-2f31-4bd4-bfd7-9286e7620151-istiod-ca-cert\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.454308 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-token\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.454409 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-envoy\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.454502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.454501 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-podinfo\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.454560 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-podinfo\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.454735 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-credential-socket\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.454829 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-credential-socket\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.454881 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-certs\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.455061 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-token" pod "92a204c6-cc2c-40ff-8c62-e5648d4280e1" (UID: "92a204c6-cc2c-40ff-8c62-e5648d4280e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.455665 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zskqs\" (UniqueName: \"kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-kube-api-access-zskqs\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.455750 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istiod-ca-cert\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.456000 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.456024 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.456061 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.456081 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w225v\" (UniqueName: \"kubernetes.io/projected/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-kube-api-access-w225v\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.456096 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.456128 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.456145 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.456159 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.456374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.456174 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b1ba1d2-444c-4dfc-adea-6458e9e4c590-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.457417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.457188 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istiod-ca-cert\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.457678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.457657 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-data\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.457863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.457836 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-data" pod "92a204c6-cc2c-40ff-8c62-e5648d4280e1" (UID: "92a204c6-cc2c-40ff-8c62-e5648d4280e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.458070 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.458036 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-certs" pod "92a204c6-cc2c-40ff-8c62-e5648d4280e1" (UID: "92a204c6-cc2c-40ff-8c62-e5648d4280e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.458171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.458126 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-credential-socket\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.458388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.458362 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-socket\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.458388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.458375 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-credential-socket\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.459542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.458543 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-data\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.459542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.458971 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-socket\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.459542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.458982 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-certs\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.459542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.459180 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60eb6552-2f31-4bd4-bfd7-9286e7620151-istiod-ca-cert\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.459542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.459321 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-certs\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.460045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.459696 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-podinfo" pod "92a204c6-cc2c-40ff-8c62-e5648d4280e1" (UID: "92a204c6-cc2c-40ff-8c62-e5648d4280e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:06.461845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.461821 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-podinfo\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.462827 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.462791 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-podinfo\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.463216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.462989 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-envoy\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.463216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.463093 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-kube-api-access-qxnl8" pod "92a204c6-cc2c-40ff-8c62-e5648d4280e1" (UID: "92a204c6-cc2c-40ff-8c62-e5648d4280e1"). InnerVolumeSpecName "kube-api-access-qxnl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:06.463298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.463266 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-envoy\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.464365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.464340 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-envoy" pod "92a204c6-cc2c-40ff-8c62-e5648d4280e1" (UID: "92a204c6-cc2c-40ff-8c62-e5648d4280e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:06.473537 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.473511 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw5l2\" (UniqueName: \"kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-kube-api-access-mw5l2\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.479368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.479342 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-token\") pod \"e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.484979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.484951 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-token\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.485207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.485179 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zskqs\" (UniqueName: \"kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-kube-api-access-zskqs\") pod \"e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.558416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.558290 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.563589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.560212 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:06.577497 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.577471 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxnl8\" (UniqueName: \"kubernetes.io/projected/92a204c6-cc2c-40ff-8c62-e5648d4280e1-kube-api-access-qxnl8\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.577701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.577681 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.577819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.577806 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.577917 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.577906 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a204c6-cc2c-40ff-8c62-e5648d4280e1-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:06.615477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.615446 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d"] Oct 31 13:44:06.623313 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.623272 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-btf5d"] Oct 31 13:44:06.636753 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.636724 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5"] Oct 31 13:44:06.647924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.647897 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:06.649862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.649832 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-7vfs5"] Oct 31 13:44:06.897863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.897837 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22"] Oct 31 13:44:06.898912 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:44:06.898885 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab664d5_8c33_46a2_a92d_aa8ebba3a743.slice/crio-0643cad9b09a5629b8e1f4833255938097de41927ee2d9d70b9d0234e6822859 WatchSource:0}: Error finding container 0643cad9b09a5629b8e1f4833255938097de41927ee2d9d70b9d0234e6822859: Status 404 returned error can't find the container with id 0643cad9b09a5629b8e1f4833255938097de41927ee2d9d70b9d0234e6822859 Oct 31 13:44:06.902009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.901980 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:44:06.902080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.902045 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:44:06.902080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.902077 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:44:06.981023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.980820 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz"] Oct 31 13:44:06.982309 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:44:06.982271 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60eb6552_2f31_4bd4_bfd7_9286e7620151.slice/crio-b79935bc111ba4c6bb25524b1111a304a9f0b1de43d5df1139bbabd629562ebe WatchSource:0}: Error finding container b79935bc111ba4c6bb25524b1111a304a9f0b1de43d5df1139bbabd629562ebe: Status 404 returned error can't find the container with id b79935bc111ba4c6bb25524b1111a304a9f0b1de43d5df1139bbabd629562ebe Oct 31 13:44:06.985645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.985607 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:44:06.985745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.985680 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:44:06.985745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:06.985712 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:44:07.280148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:07.280107 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" event={"ID":"92a204c6-cc2c-40ff-8c62-e5648d4280e1","Type":"ContainerDied","Data":"e25e41f91cb7265dd3b0981f7b52ad28f5c7d703edaef684d090ca0fed34c8c8"} Oct 31 13:44:07.280617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:07.280192 5532 scope.go:122] "RemoveContainer" containerID="0f1abfd9ff5a19595d96eb79954e4d0a54033163e37e4f6c69353569ba03cc61" Oct 31 13:44:07.280718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:07.280654 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j" Oct 31 13:44:07.285215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:07.285188 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" event={"ID":"7ab664d5-8c33-46a2-a92d-aa8ebba3a743","Type":"ContainerStarted","Data":"c137e12a4c2ccf25f895bccc66f5283d3bc7bd11f4c43d7afd23c59ec60d0077"} Oct 31 13:44:07.285324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:07.285239 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" event={"ID":"7ab664d5-8c33-46a2-a92d-aa8ebba3a743","Type":"ContainerStarted","Data":"0643cad9b09a5629b8e1f4833255938097de41927ee2d9d70b9d0234e6822859"} Oct 31 13:44:07.299903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:07.287778 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" event={"ID":"60eb6552-2f31-4bd4-bfd7-9286e7620151","Type":"ContainerStarted","Data":"c3432d0d9aa6508df3debcab6523d69add315abc4d78c18e7e715c63ebe92522"} Oct 31 13:44:07.299903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:07.287831 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" event={"ID":"60eb6552-2f31-4bd4-bfd7-9286e7620151","Type":"ContainerStarted","Data":"b79935bc111ba4c6bb25524b1111a304a9f0b1de43d5df1139bbabd629562ebe"} Oct 31 13:44:07.342042 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:07.341940 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" podStartSLOduration=254.341926742 podStartE2EDuration="4m14.341926742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:44:07.340377587 +0000 UTC m=+2415.486121429" watchObservedRunningTime="2026-10-31 13:44:07.341926742 +0000 UTC m=+2415.487670628" Oct 31 13:44:07.372828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:07.372780 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" podStartSLOduration=256.372764912 podStartE2EDuration="4m16.372764912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:44:07.370934514 +0000 UTC m=+2415.516678343" watchObservedRunningTime="2026-10-31 13:44:07.372764912 +0000 UTC m=+2415.518508740" Oct 31 13:44:07.398197 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:07.397674 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j"] Oct 31 13:44:07.410825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:07.410798 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-5qw7j"] Oct 31 13:44:07.560520 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:07.560493 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:07.649497 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:07.649461 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:08.006611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:08.006521 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b1ba1d2-444c-4dfc-adea-6458e9e4c590" path="/var/lib/kubelet/pods/2b1ba1d2-444c-4dfc-adea-6458e9e4c590/volumes" Oct 31 13:44:08.007249 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:08.007231 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bb8d480-1daf-41eb-aab7-e988dc35c2d6" path="/var/lib/kubelet/pods/7bb8d480-1daf-41eb-aab7-e988dc35c2d6/volumes" Oct 31 13:44:08.007782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:08.007768 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a204c6-cc2c-40ff-8c62-e5648d4280e1" path="/var/lib/kubelet/pods/92a204c6-cc2c-40ff-8c62-e5648d4280e1/volumes" Oct 31 13:44:08.566092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:08.566051 5532 patch_prober.go:29] interesting pod/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 13:44:08.566630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:08.566111 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" podUID="7ab664d5-8c33-46a2-a92d-aa8ebba3a743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 13:44:08.670308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:08.670260 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:09.331625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:09.331579 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:09.333400 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:09.332673 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:44:09.564525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:09.564489 5532 patch_prober.go:29] interesting pod/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 13:44:09.564945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:09.564913 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" podUID="7ab664d5-8c33-46a2-a92d-aa8ebba3a743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 13:44:10.564799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:10.564754 5532 patch_prober.go:29] interesting pod/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 13:44:10.565298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:10.564835 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" podUID="7ab664d5-8c33-46a2-a92d-aa8ebba3a743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 13:44:11.581651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:11.576619 5532 patch_prober.go:29] interesting pod/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 13:44:11.581651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:11.576683 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" podUID="7ab664d5-8c33-46a2-a92d-aa8ebba3a743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 13:44:12.298534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:12.298497 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg"] Oct 31 13:44:12.298785 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:12.298757 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" podUID="68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" containerName="istio-proxy" containerID="cri-o://befdafdde9821cbd61f761ea6758e7a17197ff65cb4ef37dcd91e34f294e9875" gracePeriod=30 Oct 31 13:44:12.576700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:12.570724 5532 patch_prober.go:29] interesting pod/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 13:44:12.576700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:12.570791 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" podUID="7ab664d5-8c33-46a2-a92d-aa8ebba3a743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 13:44:13.561802 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:13.561761 5532 patch_prober.go:29] interesting pod/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body= Oct 31 13:44:13.562327 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:13.561856 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" podUID="7ab664d5-8c33-46a2-a92d-aa8ebba3a743" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" Oct 31 13:44:14.571898 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:14.571860 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:14.572691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:14.572662 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:14.573703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:14.573670 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:15.808546 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:15.808510 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb"] Oct 31 13:44:15.809025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:15.808762 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" containerID="cri-o://14a48974c3b3a300219f53a4b7f3fe489c4e899c58f4501dd66ff06f0419e044" gracePeriod=30 Oct 31 13:44:17.442812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.442781 5532 generic.go:356] "Generic (PLEG): container finished" podID="68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" containerID="befdafdde9821cbd61f761ea6758e7a17197ff65cb4ef37dcd91e34f294e9875" exitCode=0 Oct 31 13:44:17.443202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.442862 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" event={"ID":"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758","Type":"ContainerDied","Data":"befdafdde9821cbd61f761ea6758e7a17197ff65cb4ef37dcd91e34f294e9875"} Oct 31 13:44:17.622370 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.622333 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:44:17.711919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.711881 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-socket\") pod \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " Oct 31 13:44:17.712114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.711967 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-certs\") pod \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " Oct 31 13:44:17.712114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.712001 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istiod-ca-cert\") pod \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " Oct 31 13:44:17.712256 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.712208 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-token\" (UniqueName: \"kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-token\") pod \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " Oct 31 13:44:17.713346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.712367 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-data\") pod \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " Oct 31 13:44:17.713346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.712400 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-envoy\") pod \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " Oct 31 13:44:17.713346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.712388 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-certs" pod "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" (UID: "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.713346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.712440 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-podinfo\") pod \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " Oct 31 13:44:17.713346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.712613 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-credential-socket\") pod \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " Oct 31 13:44:17.713346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.712698 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-data" pod "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" (UID: "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.713346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.712726 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-kube-api-access-h7zns\" (UniqueName: \"kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-kube-api-access-h7zns\") pod \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\" (UID: \"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758\") " Oct 31 13:44:17.713346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.713326 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-socket" pod "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" (UID: "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.713859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.713513 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-credential-socket" pod "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" (UID: "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.713859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.713820 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.713859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.713844 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.714003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.713860 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.714003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.713874 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.714433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.714396 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istiod-ca-cert" pod "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" (UID: "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:17.717965 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.717937 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-podinfo" pod "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" (UID: "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:17.718127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.718095 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-token" pod "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" (UID: "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:17.719082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.719051 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-envoy" pod "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" (UID: "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:17.719689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.719665 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-kube-api-access-h7zns" pod "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" (UID: "68d4a1f6-d0cd-4ce9-8ea6-68c6550da758"). InnerVolumeSpecName "kube-api-access-h7zns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:17.815344 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.815304 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.815344 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.815339 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.815344 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.815349 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.815704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.815359 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:17.815704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:17.815369 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7zns\" (UniqueName: \"kubernetes.io/projected/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758-kube-api-access-h7zns\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:18.360380 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:18.360290 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:44:18.453234 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:18.453209 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" Oct 31 13:44:18.453666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:18.453231 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg" event={"ID":"68d4a1f6-d0cd-4ce9-8ea6-68c6550da758","Type":"ContainerDied","Data":"7f2c8aa18e36d51137818b9806e373c4d05bf26399df9598f14886e41f5b9ff9"} Oct 31 13:44:18.453666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:18.453289 5532 scope.go:122] "RemoveContainer" containerID="befdafdde9821cbd61f761ea6758e7a17197ff65cb4ef37dcd91e34f294e9875" Oct 31 13:44:18.497629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:18.497581 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg"] Oct 31 13:44:18.503226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:18.503204 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-q7zxg"] Oct 31 13:44:20.071893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:20.071853 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" path="/var/lib/kubelet/pods/68d4a1f6-d0cd-4ce9-8ea6-68c6550da758/volumes" Oct 31 13:44:21.131609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.131568 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:44:21.142927 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.142901 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-credential-socket\") pod \"761abd89-b630-42f8-91bc-6096ef23d6a4\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " Oct 31 13:44:21.143036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.143000 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/761abd89-b630-42f8-91bc-6096ef23d6a4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761abd89-b630-42f8-91bc-6096ef23d6a4-istiod-ca-cert\") pod \"761abd89-b630-42f8-91bc-6096ef23d6a4\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " Oct 31 13:44:21.143071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.143051 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-envoy\") pod \"761abd89-b630-42f8-91bc-6096ef23d6a4\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " Oct 31 13:44:21.143124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.143111 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-socket\") pod \"761abd89-b630-42f8-91bc-6096ef23d6a4\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " Oct 31 13:44:21.143159 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.143132 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-certs\") pod \"761abd89-b630-42f8-91bc-6096ef23d6a4\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " Oct 31 13:44:21.143159 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.143145 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-credential-socket" pod "761abd89-b630-42f8-91bc-6096ef23d6a4" (UID: "761abd89-b630-42f8-91bc-6096ef23d6a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.143292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.143276 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-token\" (UniqueName: \"kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-token\") pod \"761abd89-b630-42f8-91bc-6096ef23d6a4\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " Oct 31 13:44:21.143384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.143371 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-kube-api-access-fd574\" (UniqueName: \"kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-kube-api-access-fd574\") pod \"761abd89-b630-42f8-91bc-6096ef23d6a4\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " Oct 31 13:44:21.143451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.143394 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-certs" pod "761abd89-b630-42f8-91bc-6096ef23d6a4" (UID: "761abd89-b630-42f8-91bc-6096ef23d6a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.143451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.143414 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-socket" pod "761abd89-b630-42f8-91bc-6096ef23d6a4" (UID: "761abd89-b630-42f8-91bc-6096ef23d6a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.143451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.143427 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761abd89-b630-42f8-91bc-6096ef23d6a4-istiod-ca-cert" pod "761abd89-b630-42f8-91bc-6096ef23d6a4" (UID: "761abd89-b630-42f8-91bc-6096ef23d6a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:21.143451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.143433 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-podinfo\") pod \"761abd89-b630-42f8-91bc-6096ef23d6a4\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " Oct 31 13:44:21.143715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.143510 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-data\") pod \"761abd89-b630-42f8-91bc-6096ef23d6a4\" (UID: \"761abd89-b630-42f8-91bc-6096ef23d6a4\") " Oct 31 13:44:21.143821 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.143800 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-data" pod "761abd89-b630-42f8-91bc-6096ef23d6a4" (UID: "761abd89-b630-42f8-91bc-6096ef23d6a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.144351 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.144329 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.144441 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.144350 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761abd89-b630-42f8-91bc-6096ef23d6a4-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.144441 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.144376 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.144441 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.144391 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.144441 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.144403 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.145554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.145528 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-kube-api-access-fd574" pod "761abd89-b630-42f8-91bc-6096ef23d6a4" (UID: "761abd89-b630-42f8-91bc-6096ef23d6a4"). InnerVolumeSpecName "kube-api-access-fd574". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:21.145651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.145572 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-token" pod "761abd89-b630-42f8-91bc-6096ef23d6a4" (UID: "761abd89-b630-42f8-91bc-6096ef23d6a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:21.145830 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.145797 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-envoy" pod "761abd89-b630-42f8-91bc-6096ef23d6a4" (UID: "761abd89-b630-42f8-91bc-6096ef23d6a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:21.145830 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.145801 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-podinfo" pod "761abd89-b630-42f8-91bc-6096ef23d6a4" (UID: "761abd89-b630-42f8-91bc-6096ef23d6a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:21.246810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.246774 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.246810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.246809 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd574\" (UniqueName: \"kubernetes.io/projected/761abd89-b630-42f8-91bc-6096ef23d6a4-kube-api-access-fd574\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.246810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.246820 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.247098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.246830 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761abd89-b630-42f8-91bc-6096ef23d6a4-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:21.484676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.484638 5532 generic.go:356] "Generic (PLEG): container finished" podID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerID="14a48974c3b3a300219f53a4b7f3fe489c4e899c58f4501dd66ff06f0419e044" exitCode=0 Oct 31 13:44:21.484858 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.484749 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" Oct 31 13:44:21.484858 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.484755 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" event={"ID":"761abd89-b630-42f8-91bc-6096ef23d6a4","Type":"ContainerDied","Data":"14a48974c3b3a300219f53a4b7f3fe489c4e899c58f4501dd66ff06f0419e044"} Oct 31 13:44:21.484858 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.484842 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb" event={"ID":"761abd89-b630-42f8-91bc-6096ef23d6a4","Type":"ContainerDied","Data":"ce6f5ea5165bc2e7a49d95e234b77738dff2587372c0b72c0bbe5c43686fa108"} Oct 31 13:44:21.484964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.484866 5532 scope.go:122] "RemoveContainer" containerID="14a48974c3b3a300219f53a4b7f3fe489c4e899c58f4501dd66ff06f0419e044" Oct 31 13:44:21.497331 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.497312 5532 scope.go:122] "RemoveContainer" containerID="14a48974c3b3a300219f53a4b7f3fe489c4e899c58f4501dd66ff06f0419e044" Oct 31 13:44:21.497662 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:44:21.497589 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a48974c3b3a300219f53a4b7f3fe489c4e899c58f4501dd66ff06f0419e044\": container with ID starting with 14a48974c3b3a300219f53a4b7f3fe489c4e899c58f4501dd66ff06f0419e044 not found: ID does not exist" containerID="14a48974c3b3a300219f53a4b7f3fe489c4e899c58f4501dd66ff06f0419e044" Oct 31 13:44:21.497662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.497629 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a48974c3b3a300219f53a4b7f3fe489c4e899c58f4501dd66ff06f0419e044"} err="failed to get container status \"14a48974c3b3a300219f53a4b7f3fe489c4e899c58f4501dd66ff06f0419e044\": rpc error: code = NotFound desc = could not find container \"14a48974c3b3a300219f53a4b7f3fe489c4e899c58f4501dd66ff06f0419e044\": container with ID starting with 14a48974c3b3a300219f53a4b7f3fe489c4e899c58f4501dd66ff06f0419e044 not found: ID does not exist" Oct 31 13:44:21.512983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.512960 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb"] Oct 31 13:44:21.519983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:21.519961 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-gv9wb"] Oct 31 13:44:22.046929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:22.046894 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" path="/var/lib/kubelet/pods/761abd89-b630-42f8-91bc-6096ef23d6a4/volumes" Oct 31 13:44:31.764032 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:44:31.763949 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:44:32.512878 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:32.512836 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22"] Oct 31 13:44:32.513205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:32.513142 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" podUID="7ab664d5-8c33-46a2-a92d-aa8ebba3a743" containerName="istio-proxy" containerID="cri-o://c137e12a4c2ccf25f895bccc66f5283d3bc7bd11f4c43d7afd23c59ec60d0077" gracePeriod=30 Oct 31 13:44:37.739624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.735665 5532 generic.go:356] "Generic (PLEG): container finished" podID="7ab664d5-8c33-46a2-a92d-aa8ebba3a743" containerID="c137e12a4c2ccf25f895bccc66f5283d3bc7bd11f4c43d7afd23c59ec60d0077" exitCode=0 Oct 31 13:44:37.739624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.735758 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" event={"ID":"7ab664d5-8c33-46a2-a92d-aa8ebba3a743","Type":"ContainerDied","Data":"c137e12a4c2ccf25f895bccc66f5283d3bc7bd11f4c43d7afd23c59ec60d0077"} Oct 31 13:44:37.850941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.850910 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:37.944737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.944701 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-envoy\") pod \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " Oct 31 13:44:37.944921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.944823 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-certs\") pod \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " Oct 31 13:44:37.945091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.945060 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-credential-socket\") pod \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " Oct 31 13:44:37.945187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.945117 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-certs" pod "7ab664d5-8c33-46a2-a92d-aa8ebba3a743" (UID: "7ab664d5-8c33-46a2-a92d-aa8ebba3a743"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:37.945187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.945132 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-socket\") pod \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " Oct 31 13:44:37.945321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.945241 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istiod-ca-cert\") pod \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " Oct 31 13:44:37.945431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.945384 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-podinfo\") pod \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " Oct 31 13:44:37.945513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.945385 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-socket" pod "7ab664d5-8c33-46a2-a92d-aa8ebba3a743" (UID: "7ab664d5-8c33-46a2-a92d-aa8ebba3a743"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:37.945513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.945458 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-kube-api-access-mw5l2\" (UniqueName: \"kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-kube-api-access-mw5l2\") pod \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " Oct 31 13:44:37.945635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.945537 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-token\" (UniqueName: \"kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-token\") pod \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " Oct 31 13:44:37.945696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.945639 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-data\") pod \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\" (UID: \"7ab664d5-8c33-46a2-a92d-aa8ebba3a743\") " Oct 31 13:44:37.946084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.945430 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-credential-socket" pod "7ab664d5-8c33-46a2-a92d-aa8ebba3a743" (UID: "7ab664d5-8c33-46a2-a92d-aa8ebba3a743"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:37.946084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.945815 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istiod-ca-cert" pod "7ab664d5-8c33-46a2-a92d-aa8ebba3a743" (UID: "7ab664d5-8c33-46a2-a92d-aa8ebba3a743"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:37.946227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.946158 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-data" pod "7ab664d5-8c33-46a2-a92d-aa8ebba3a743" (UID: "7ab664d5-8c33-46a2-a92d-aa8ebba3a743"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:37.946663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.946636 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:37.946778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.946664 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:37.946778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.946679 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:37.946778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.946693 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:37.946778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.946709 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:37.949065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.948634 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-podinfo" pod "7ab664d5-8c33-46a2-a92d-aa8ebba3a743" (UID: "7ab664d5-8c33-46a2-a92d-aa8ebba3a743"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:37.949065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.948707 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-token" pod "7ab664d5-8c33-46a2-a92d-aa8ebba3a743" (UID: "7ab664d5-8c33-46a2-a92d-aa8ebba3a743"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:37.949065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.949003 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-kube-api-access-mw5l2" pod "7ab664d5-8c33-46a2-a92d-aa8ebba3a743" (UID: "7ab664d5-8c33-46a2-a92d-aa8ebba3a743"). InnerVolumeSpecName "kube-api-access-mw5l2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:37.949759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:37.949735 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-envoy" pod "7ab664d5-8c33-46a2-a92d-aa8ebba3a743" (UID: "7ab664d5-8c33-46a2-a92d-aa8ebba3a743"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:38.048587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:38.048559 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:38.048587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:38.048585 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:38.048828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:38.048620 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:38.048828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:38.048632 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw5l2\" (UniqueName: \"kubernetes.io/projected/7ab664d5-8c33-46a2-a92d-aa8ebba3a743-kube-api-access-mw5l2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:38.746023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:38.745990 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" Oct 31 13:44:38.746476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:38.746022 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22" event={"ID":"7ab664d5-8c33-46a2-a92d-aa8ebba3a743","Type":"ContainerDied","Data":"0643cad9b09a5629b8e1f4833255938097de41927ee2d9d70b9d0234e6822859"} Oct 31 13:44:38.746476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:38.746092 5532 scope.go:122] "RemoveContainer" containerID="c137e12a4c2ccf25f895bccc66f5283d3bc7bd11f4c43d7afd23c59ec60d0077" Oct 31 13:44:38.775730 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:38.775697 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22"] Oct 31 13:44:38.783869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:38.783846 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-69bdf5-openshift-default-784459fbf9-5rz22"] Oct 31 13:44:40.006910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:40.006876 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab664d5-8c33-46a2-a92d-aa8ebba3a743" path="/var/lib/kubelet/pods/7ab664d5-8c33-46a2-a92d-aa8ebba3a743/volumes" Oct 31 13:44:41.570024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:41.569991 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw"] Oct 31 13:44:41.570486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:41.570243 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" containerID="cri-o://b31d278fad14c28399cb16b358c33b520b368ac30e09465ffa19384b7f4e1838" gracePeriod=30 Oct 31 13:44:42.078669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:42.078629 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-srpzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:44:42.078835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:42.078692 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:44:45.048609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.048558 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh"] Oct 31 13:44:45.049521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.049503 5532 cpu_manager.go:414] "containerMap: removing container" podUID="7ab664d5-8c33-46a2-a92d-aa8ebba3a743" containerName="istio-proxy" Oct 31 13:44:45.049566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.049522 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab664d5-8c33-46a2-a92d-aa8ebba3a743" containerName="istio-proxy" Oct 31 13:44:45.049566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.049538 5532 cpu_manager.go:414] "containerMap: removing container" podUID="68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" containerName="istio-proxy" Oct 31 13:44:45.049566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.049544 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" containerName="istio-proxy" Oct 31 13:44:45.049680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.049577 5532 cpu_manager.go:414] "containerMap: removing container" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" Oct 31 13:44:45.049680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.049583 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" Oct 31 13:44:45.049680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.049616 5532 cpu_manager.go:414] "containerMap: removing container" podUID="92a204c6-cc2c-40ff-8c62-e5648d4280e1" containerName="istio-proxy" Oct 31 13:44:45.049680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.049621 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a204c6-cc2c-40ff-8c62-e5648d4280e1" containerName="istio-proxy" Oct 31 13:44:45.049806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.049793 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab664d5-8c33-46a2-a92d-aa8ebba3a743" containerName="istio-proxy" Oct 31 13:44:45.049806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.049802 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d4a1f6-d0cd-4ce9-8ea6-68c6550da758" containerName="istio-proxy" Oct 31 13:44:45.049869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.049815 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="761abd89-b630-42f8-91bc-6096ef23d6a4" containerName="istio-proxy" Oct 31 13:44:45.049869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.049822 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a204c6-cc2c-40ff-8c62-e5648d4280e1" containerName="istio-proxy" Oct 31 13:44:45.055773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.055737 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" Oct 31 13:44:45.060353 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.060323 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh"] Oct 31 13:44:45.064748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.064716 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:44:45.064952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.064928 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:44:45.216532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.211107 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg628\" (UniqueName: \"kubernetes.io/projected/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-kube-api-access-dg628\") pod \"maas-api-revoke-sub-bd7417ec-v6qbh\" (UID: \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" Oct 31 13:44:45.216532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.211208 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd7417ec-v6qbh\" (UID: \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" Oct 31 13:44:45.216532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.211265 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd7417ec-v6qbh\" (UID: \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" Oct 31 13:44:45.313808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.313699 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg628\" (UniqueName: \"kubernetes.io/projected/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-kube-api-access-dg628\") pod \"maas-api-revoke-sub-bd7417ec-v6qbh\" (UID: \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" Oct 31 13:44:45.313973 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.313814 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd7417ec-v6qbh\" (UID: \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" Oct 31 13:44:45.313973 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.313850 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd7417ec-v6qbh\" (UID: \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" Oct 31 13:44:45.314566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.314538 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd7417ec-v6qbh\" (UID: \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" Oct 31 13:44:45.316518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.316497 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd7417ec-v6qbh\" (UID: \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" Oct 31 13:44:45.327138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.327115 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg628\" (UniqueName: \"kubernetes.io/projected/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-kube-api-access-dg628\") pod \"maas-api-revoke-sub-bd7417ec-v6qbh\" (UID: \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" Oct 31 13:44:45.380217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.380183 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" Oct 31 13:44:45.691454 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:44:45.691426 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba2749d_8c41_4622_abe7_33cd7ed5aef5.slice/crio-82de140fe9ff1b9e845a746f3e5c1499b11d1d0915208566c1d23d01e21d14a2 WatchSource:0}: Error finding container 82de140fe9ff1b9e845a746f3e5c1499b11d1d0915208566c1d23d01e21d14a2: Status 404 returned error can't find the container with id 82de140fe9ff1b9e845a746f3e5c1499b11d1d0915208566c1d23d01e21d14a2 Oct 31 13:44:45.692294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.692267 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh"] Oct 31 13:44:45.821510 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:45.821468 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" event={"ID":"8ba2749d-8c41-4622-abe7-33cd7ed5aef5","Type":"ContainerStarted","Data":"82de140fe9ff1b9e845a746f3e5c1499b11d1d0915208566c1d23d01e21d14a2"} Oct 31 13:44:46.854439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.854404 5532 generic.go:356] "Generic (PLEG): container finished" podID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerID="b31d278fad14c28399cb16b358c33b520b368ac30e09465ffa19384b7f4e1838" exitCode=0 Oct 31 13:44:46.854840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.854475 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" event={"ID":"3fb7e7a8-5247-4e9d-966a-7a4be7c06673","Type":"ContainerDied","Data":"b31d278fad14c28399cb16b358c33b520b368ac30e09465ffa19384b7f4e1838"} Oct 31 13:44:46.856560 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.856532 5532 generic.go:356] "Generic (PLEG): container finished" podID="8ba2749d-8c41-4622-abe7-33cd7ed5aef5" containerID="b6004adca5ca1534c6680b4d0b0ff32746de63cc1d76731a171ea9dccb443679" exitCode=0 Oct 31 13:44:46.856737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.856623 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" event={"ID":"8ba2749d-8c41-4622-abe7-33cd7ed5aef5","Type":"ContainerDied","Data":"b6004adca5ca1534c6680b4d0b0ff32746de63cc1d76731a171ea9dccb443679"} Oct 31 13:44:46.888999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.888973 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:44:46.953340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.953301 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istiod-ca-cert\") pod \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " Oct 31 13:44:46.954349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.954321 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-token\" (UniqueName: \"kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-token\") pod \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " Oct 31 13:44:46.954658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.954563 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-certs\") pod \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " Oct 31 13:44:46.954789 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.954111 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istiod-ca-cert" pod "3fb7e7a8-5247-4e9d-966a-7a4be7c06673" (UID: "3fb7e7a8-5247-4e9d-966a-7a4be7c06673"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:46.955104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.955001 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-certs" pod "3fb7e7a8-5247-4e9d-966a-7a4be7c06673" (UID: "3fb7e7a8-5247-4e9d-966a-7a4be7c06673"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:46.955104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.955115 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-socket\") pod \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " Oct 31 13:44:46.955287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.955225 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-credential-socket\") pod \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " Oct 31 13:44:46.955347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.955332 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-kube-api-access-vj4ht\" (UniqueName: \"kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-kube-api-access-vj4ht\") pod \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " Oct 31 13:44:46.955405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.955361 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-envoy\") pod \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " Oct 31 13:44:46.955405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.955400 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-podinfo\") pod \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " Oct 31 13:44:46.955527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.955505 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-data\") pod \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\" (UID: \"3fb7e7a8-5247-4e9d-966a-7a4be7c06673\") " Oct 31 13:44:46.956844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.956443 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:46.956844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.956501 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:46.957338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.957219 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-socket" pod "3fb7e7a8-5247-4e9d-966a-7a4be7c06673" (UID: "3fb7e7a8-5247-4e9d-966a-7a4be7c06673"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:46.957338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.957285 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-credential-socket" pod "3fb7e7a8-5247-4e9d-966a-7a4be7c06673" (UID: "3fb7e7a8-5247-4e9d-966a-7a4be7c06673"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:46.957763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.957526 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-token" pod "3fb7e7a8-5247-4e9d-966a-7a4be7c06673" (UID: "3fb7e7a8-5247-4e9d-966a-7a4be7c06673"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:46.957763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.957699 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-data" pod "3fb7e7a8-5247-4e9d-966a-7a4be7c06673" (UID: "3fb7e7a8-5247-4e9d-966a-7a4be7c06673"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:46.964175 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.964134 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-podinfo" pod "3fb7e7a8-5247-4e9d-966a-7a4be7c06673" (UID: "3fb7e7a8-5247-4e9d-966a-7a4be7c06673"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:44:46.966707 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.965002 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-envoy" pod "3fb7e7a8-5247-4e9d-966a-7a4be7c06673" (UID: "3fb7e7a8-5247-4e9d-966a-7a4be7c06673"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:44:46.966707 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:46.965075 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-kube-api-access-vj4ht" pod "3fb7e7a8-5247-4e9d-966a-7a4be7c06673" (UID: "3fb7e7a8-5247-4e9d-966a-7a4be7c06673"). InnerVolumeSpecName "kube-api-access-vj4ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:47.058405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:47.058366 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:47.058405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:47.058403 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:47.058647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:47.058418 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:47.058647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:47.058434 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj4ht\" (UniqueName: \"kubernetes.io/projected/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-kube-api-access-vj4ht\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:47.058647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:47.058450 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:47.058647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:47.058464 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:47.058647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:47.058477 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fb7e7a8-5247-4e9d-966a-7a4be7c06673-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:47.867041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:47.867012 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" Oct 31 13:44:47.867524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:47.867040 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw" event={"ID":"3fb7e7a8-5247-4e9d-966a-7a4be7c06673","Type":"ContainerDied","Data":"339de1adf565ff16192d4821805408550a5f9c0f0d1c4897a7a6f013fd4503f9"} Oct 31 13:44:47.867524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:47.867104 5532 scope.go:122] "RemoveContainer" containerID="b31d278fad14c28399cb16b358c33b520b368ac30e09465ffa19384b7f4e1838" Oct 31 13:44:47.934665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:47.906652 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw"] Oct 31 13:44:47.942116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:47.942087 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-srpzw"] Oct 31 13:44:48.025632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.024852 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" path="/var/lib/kubelet/pods/3fb7e7a8-5247-4e9d-966a-7a4be7c06673/volumes" Oct 31 13:44:48.100378 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.100349 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" Oct 31 13:44:48.187261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.187227 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-kube-api-access-dg628\" (UniqueName: \"kubernetes.io/projected/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-kube-api-access-dg628\") pod \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\" (UID: \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\") " Oct 31 13:44:48.187458 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.187376 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-openshift-ca\") pod \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\" (UID: \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\") " Oct 31 13:44:48.187747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.187723 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-openshift-ca" pod "8ba2749d-8c41-4622-abe7-33cd7ed5aef5" (UID: "8ba2749d-8c41-4622-abe7-33cd7ed5aef5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:48.187834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.187789 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-xks-ca\") pod \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\" (UID: \"8ba2749d-8c41-4622-abe7-33cd7ed5aef5\") " Oct 31 13:44:48.188690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.188664 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:48.189579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.189550 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-kube-api-access-dg628" pod "8ba2749d-8c41-4622-abe7-33cd7ed5aef5" (UID: "8ba2749d-8c41-4622-abe7-33cd7ed5aef5"). InnerVolumeSpecName "kube-api-access-dg628". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:48.189817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.189792 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-xks-ca" pod "8ba2749d-8c41-4622-abe7-33cd7ed5aef5" (UID: "8ba2749d-8c41-4622-abe7-33cd7ed5aef5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:48.290265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.290230 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:48.290265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.290259 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg628\" (UniqueName: \"kubernetes.io/projected/8ba2749d-8c41-4622-abe7-33cd7ed5aef5-kube-api-access-dg628\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:48.880437 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.880406 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" Oct 31 13:44:48.880437 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.880425 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh" event={"ID":"8ba2749d-8c41-4622-abe7-33cd7ed5aef5","Type":"ContainerDied","Data":"82de140fe9ff1b9e845a746f3e5c1499b11d1d0915208566c1d23d01e21d14a2"} Oct 31 13:44:48.880437 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:48.880499 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82de140fe9ff1b9e845a746f3e5c1499b11d1d0915208566c1d23d01e21d14a2" Oct 31 13:44:56.275990 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.275957 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7"] Oct 31 13:44:56.277909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.277879 5532 cpu_manager.go:414] "containerMap: removing container" podUID="8ba2749d-8c41-4622-abe7-33cd7ed5aef5" containerName="revoke-keys" Oct 31 13:44:56.277909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.277913 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ba2749d-8c41-4622-abe7-33cd7ed5aef5" containerName="revoke-keys" Oct 31 13:44:56.278124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.277982 5532 cpu_manager.go:414] "containerMap: removing container" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" Oct 31 13:44:56.278124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.277991 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" Oct 31 13:44:56.278605 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.278574 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ba2749d-8c41-4622-abe7-33cd7ed5aef5" containerName="revoke-keys" Oct 31 13:44:56.278704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.278627 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fb7e7a8-5247-4e9d-966a-7a4be7c06673" containerName="istio-proxy" Oct 31 13:44:56.287964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.287937 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" Oct 31 13:44:56.295157 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.295131 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:44:56.297141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.296278 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7"] Oct 31 13:44:56.297329 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.296312 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:44:56.361070 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.361038 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d74d07cc-vv9n7\" (UID: \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" Oct 31 13:44:56.361233 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.361082 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfngh\" (UniqueName: \"kubernetes.io/projected/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-kube-api-access-jfngh\") pod \"maas-api-revoke-sub-d74d07cc-vv9n7\" (UID: \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" Oct 31 13:44:56.361233 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.361224 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d74d07cc-vv9n7\" (UID: \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" Oct 31 13:44:56.462807 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.462767 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d74d07cc-vv9n7\" (UID: \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" Oct 31 13:44:56.462995 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.462815 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfngh\" (UniqueName: \"kubernetes.io/projected/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-kube-api-access-jfngh\") pod \"maas-api-revoke-sub-d74d07cc-vv9n7\" (UID: \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" Oct 31 13:44:56.462995 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.462921 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d74d07cc-vv9n7\" (UID: \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" Oct 31 13:44:56.463431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.463407 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d74d07cc-vv9n7\" (UID: \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" Oct 31 13:44:56.465280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.465262 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d74d07cc-vv9n7\" (UID: \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" Oct 31 13:44:56.479084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.479058 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfngh\" (UniqueName: \"kubernetes.io/projected/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-kube-api-access-jfngh\") pod \"maas-api-revoke-sub-d74d07cc-vv9n7\" (UID: \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" Oct 31 13:44:56.615879 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.615799 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" Oct 31 13:44:56.907093 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:44:56.907059 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a1f8b08_ffe4_4239_8572_bbafda1a2dde.slice/crio-9ef2f61c859d4fcde21ef759ab06132824f7ff234bae5990c1d3cf36e9c2be16 WatchSource:0}: Error finding container 9ef2f61c859d4fcde21ef759ab06132824f7ff234bae5990c1d3cf36e9c2be16: Status 404 returned error can't find the container with id 9ef2f61c859d4fcde21ef759ab06132824f7ff234bae5990c1d3cf36e9c2be16 Oct 31 13:44:56.908939 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:56.908915 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7"] Oct 31 13:44:57.043536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:57.043436 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" event={"ID":"6a1f8b08-ffe4-4239-8572-bbafda1a2dde","Type":"ContainerStarted","Data":"9ef2f61c859d4fcde21ef759ab06132824f7ff234bae5990c1d3cf36e9c2be16"} Oct 31 13:44:58.056028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:58.055992 5532 generic.go:356] "Generic (PLEG): container finished" podID="6a1f8b08-ffe4-4239-8572-bbafda1a2dde" containerID="89946794e19a6ee0a3fd63d6a029cc3990f66545896b749c51f1924b8d399888" exitCode=0 Oct 31 13:44:58.056449 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:58.056072 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" event={"ID":"6a1f8b08-ffe4-4239-8572-bbafda1a2dde","Type":"ContainerDied","Data":"89946794e19a6ee0a3fd63d6a029cc3990f66545896b749c51f1924b8d399888"} Oct 31 13:44:59.484570 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:59.484541 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" Oct 31 13:44:59.556370 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:59.556336 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-openshift-ca\") pod \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\" (UID: \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\") " Oct 31 13:44:59.556745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:59.556717 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-xks-ca\") pod \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\" (UID: \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\") " Oct 31 13:44:59.556745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:59.556730 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-openshift-ca" pod "6a1f8b08-ffe4-4239-8572-bbafda1a2dde" (UID: "6a1f8b08-ffe4-4239-8572-bbafda1a2dde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:44:59.556964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:59.556916 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-kube-api-access-jfngh\" (UniqueName: \"kubernetes.io/projected/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-kube-api-access-jfngh\") pod \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\" (UID: \"6a1f8b08-ffe4-4239-8572-bbafda1a2dde\") " Oct 31 13:44:59.557867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:59.557844 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:59.570065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:59.570036 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-xks-ca" pod "6a1f8b08-ffe4-4239-8572-bbafda1a2dde" (UID: "6a1f8b08-ffe4-4239-8572-bbafda1a2dde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:44:59.570204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:59.570092 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-kube-api-access-jfngh" pod "6a1f8b08-ffe4-4239-8572-bbafda1a2dde" (UID: "6a1f8b08-ffe4-4239-8572-bbafda1a2dde"). InnerVolumeSpecName "kube-api-access-jfngh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:44:59.659865 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:59.659830 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:44:59.659865 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:44:59.659861 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfngh\" (UniqueName: \"kubernetes.io/projected/6a1f8b08-ffe4-4239-8572-bbafda1a2dde-kube-api-access-jfngh\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:00.142250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.142213 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6"] Oct 31 13:45:00.143368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.143345 5532 cpu_manager.go:414] "containerMap: removing container" podUID="6a1f8b08-ffe4-4239-8572-bbafda1a2dde" containerName="revoke-keys" Oct 31 13:45:00.143452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.143369 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1f8b08-ffe4-4239-8572-bbafda1a2dde" containerName="revoke-keys" Oct 31 13:45:00.143565 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.143554 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1f8b08-ffe4-4239-8572-bbafda1a2dde" containerName="revoke-keys" Oct 31 13:45:00.146660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.146640 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" Oct 31 13:45:00.147723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.147695 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7" event={"ID":"6a1f8b08-ffe4-4239-8572-bbafda1a2dde","Type":"ContainerDied","Data":"9ef2f61c859d4fcde21ef759ab06132824f7ff234bae5990c1d3cf36e9c2be16"} Oct 31 13:45:00.147848 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.147729 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ef2f61c859d4fcde21ef759ab06132824f7ff234bae5990c1d3cf36e9c2be16" Oct 31 13:45:00.147848 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.147817 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" Oct 31 13:45:00.162198 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.162157 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m"] Oct 31 13:45:00.168882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.168855 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z"] Oct 31 13:45:00.169041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.168994 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" Oct 31 13:45:00.175571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.175540 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6"] Oct 31 13:45:00.175764 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.175691 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" Oct 31 13:45:00.182922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.182740 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc"] Oct 31 13:45:00.209967 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.209915 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:45:00.211994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.211964 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" Oct 31 13:45:00.224801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.224770 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z"] Oct 31 13:45:00.250879 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.250853 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m"] Oct 31 13:45:00.266678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.258964 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc"] Oct 31 13:45:00.272672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.270646 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5"] Oct 31 13:45:00.275507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.275478 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd2e824-e299-438a-ae40-d3ebc513943d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m\" (UID: \"1cd2e824-e299-438a-ae40-d3ebc513943d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" Oct 31 13:45:00.275705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.275686 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwv6l\" (UniqueName: \"kubernetes.io/projected/1cd2e824-e299-438a-ae40-d3ebc513943d-kube-api-access-qwv6l\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m\" (UID: \"1cd2e824-e299-438a-ae40-d3ebc513943d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" Oct 31 13:45:00.275769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.275738 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp2v2\" (UniqueName: \"kubernetes.io/projected/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-kube-api-access-rp2v2\") pod \"maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z\" (UID: \"e97636dc-fff6-4b9d-adf8-8d29c8becb0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" Oct 31 13:45:00.275894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.275868 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096af709-7898-469d-a55b-bbbacf1cb43e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc\" (UID: \"096af709-7898-469d-a55b-bbbacf1cb43e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" Oct 31 13:45:00.275934 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.275913 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85b42\" (UniqueName: \"kubernetes.io/projected/aba05975-872c-4c87-aff7-07080b68afb4-kube-api-access-85b42\") pod \"maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6\" (UID: \"aba05975-872c-4c87-aff7-07080b68afb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" Oct 31 13:45:00.276032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.276019 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba05975-872c-4c87-aff7-07080b68afb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6\" (UID: \"aba05975-872c-4c87-aff7-07080b68afb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" Oct 31 13:45:00.276067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.276044 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhpz2\" (UniqueName: \"kubernetes.io/projected/096af709-7898-469d-a55b-bbbacf1cb43e-kube-api-access-qhpz2\") pod \"maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc\" (UID: \"096af709-7898-469d-a55b-bbbacf1cb43e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" Oct 31 13:45:00.276193 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.276174 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z\" (UID: \"e97636dc-fff6-4b9d-adf8-8d29c8becb0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" Oct 31 13:45:00.282171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.282042 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5"] Oct 31 13:45:00.282322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.282144 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" Oct 31 13:45:00.303672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.299766 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj"] Oct 31 13:45:00.309496 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.309467 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj"] Oct 31 13:45:00.309637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.309583 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" Oct 31 13:45:00.342786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.342313 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh"] Oct 31 13:45:00.352698 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.352671 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl"] Oct 31 13:45:00.353701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.352983 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" Oct 31 13:45:00.358484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.358457 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" Oct 31 13:45:00.359509 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.359484 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh"] Oct 31 13:45:00.367544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.367370 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl"] Oct 31 13:45:00.378168 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.378113 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd2e824-e299-438a-ae40-d3ebc513943d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m\" (UID: \"1cd2e824-e299-438a-ae40-d3ebc513943d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" Oct 31 13:45:00.378352 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.378183 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkjs6\" (UniqueName: \"kubernetes.io/projected/c810f088-8712-4c24-bc79-1e19589746bb-kube-api-access-qkjs6\") pod \"maas-api-key-cleanup-29890905-5qpvj\" (UID: \"c810f088-8712-4c24-bc79-1e19589746bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" Oct 31 13:45:00.378352 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.378287 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw5pm\" (UniqueName: \"kubernetes.io/projected/b8b015f6-8071-4ff8-8269-801537622c10-kube-api-access-vw5pm\") pod \"maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5\" (UID: \"b8b015f6-8071-4ff8-8269-801537622c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" Oct 31 13:45:00.378352 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.378339 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwv6l\" (UniqueName: \"kubernetes.io/projected/1cd2e824-e299-438a-ae40-d3ebc513943d-kube-api-access-qwv6l\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m\" (UID: \"1cd2e824-e299-438a-ae40-d3ebc513943d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" Oct 31 13:45:00.378872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.378383 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp2v2\" (UniqueName: \"kubernetes.io/projected/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-kube-api-access-rp2v2\") pod \"maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z\" (UID: \"e97636dc-fff6-4b9d-adf8-8d29c8becb0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" Oct 31 13:45:00.378872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.378458 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096af709-7898-469d-a55b-bbbacf1cb43e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc\" (UID: \"096af709-7898-469d-a55b-bbbacf1cb43e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" Oct 31 13:45:00.378872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.378490 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85b42\" (UniqueName: \"kubernetes.io/projected/aba05975-872c-4c87-aff7-07080b68afb4-kube-api-access-85b42\") pod \"maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6\" (UID: \"aba05975-872c-4c87-aff7-07080b68afb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" Oct 31 13:45:00.378872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.378664 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba05975-872c-4c87-aff7-07080b68afb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6\" (UID: \"aba05975-872c-4c87-aff7-07080b68afb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" Oct 31 13:45:00.378872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.378698 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhpz2\" (UniqueName: \"kubernetes.io/projected/096af709-7898-469d-a55b-bbbacf1cb43e-kube-api-access-qhpz2\") pod \"maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc\" (UID: \"096af709-7898-469d-a55b-bbbacf1cb43e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" Oct 31 13:45:00.378872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.378729 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c810f088-8712-4c24-bc79-1e19589746bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890905-5qpvj\" (UID: \"c810f088-8712-4c24-bc79-1e19589746bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" Oct 31 13:45:00.378872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.378843 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b015f6-8071-4ff8-8269-801537622c10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5\" (UID: \"b8b015f6-8071-4ff8-8269-801537622c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" Oct 31 13:45:00.379410 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.378908 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z\" (UID: \"e97636dc-fff6-4b9d-adf8-8d29c8becb0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" Oct 31 13:45:00.379410 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.378960 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd2e824-e299-438a-ae40-d3ebc513943d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m\" (UID: \"1cd2e824-e299-438a-ae40-d3ebc513943d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" Oct 31 13:45:00.379854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.379812 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z\" (UID: \"e97636dc-fff6-4b9d-adf8-8d29c8becb0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" Oct 31 13:45:00.381190 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.381168 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096af709-7898-469d-a55b-bbbacf1cb43e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc\" (UID: \"096af709-7898-469d-a55b-bbbacf1cb43e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" Oct 31 13:45:00.381310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.381219 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp"] Oct 31 13:45:00.381674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.381657 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba05975-872c-4c87-aff7-07080b68afb4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6\" (UID: \"aba05975-872c-4c87-aff7-07080b68afb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" Oct 31 13:45:00.387676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.387659 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp"] Oct 31 13:45:00.387909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.387845 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" Oct 31 13:45:00.393420 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.393403 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69"] Oct 31 13:45:00.397945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.397924 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhpz2\" (UniqueName: \"kubernetes.io/projected/096af709-7898-469d-a55b-bbbacf1cb43e-kube-api-access-qhpz2\") pod \"maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc\" (UID: \"096af709-7898-469d-a55b-bbbacf1cb43e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" Oct 31 13:45:00.402516 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.402492 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp2v2\" (UniqueName: \"kubernetes.io/projected/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-kube-api-access-rp2v2\") pod \"maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z\" (UID: \"e97636dc-fff6-4b9d-adf8-8d29c8becb0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" Oct 31 13:45:00.405747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.405728 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85b42\" (UniqueName: \"kubernetes.io/projected/aba05975-872c-4c87-aff7-07080b68afb4-kube-api-access-85b42\") pod \"maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6\" (UID: \"aba05975-872c-4c87-aff7-07080b68afb4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" Oct 31 13:45:00.408819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.408791 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwv6l\" (UniqueName: \"kubernetes.io/projected/1cd2e824-e299-438a-ae40-d3ebc513943d-kube-api-access-qwv6l\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m\" (UID: \"1cd2e824-e299-438a-ae40-d3ebc513943d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" Oct 31 13:45:00.411756 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.411738 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69"] Oct 31 13:45:00.412027 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.411996 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" Oct 31 13:45:00.467773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.467748 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" Oct 31 13:45:00.481014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.480975 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c810f088-8712-4c24-bc79-1e19589746bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890905-5qpvj\" (UID: \"c810f088-8712-4c24-bc79-1e19589746bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" Oct 31 13:45:00.481176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.481096 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a4f033-5cb3-4d5e-998b-b66150794430-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl\" (UID: \"62a4f033-5cb3-4d5e-998b-b66150794430\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" Oct 31 13:45:00.481176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.481128 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b015f6-8071-4ff8-8269-801537622c10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5\" (UID: \"b8b015f6-8071-4ff8-8269-801537622c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" Oct 31 13:45:00.481176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.481158 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6lcm\" (UniqueName: \"kubernetes.io/projected/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-kube-api-access-t6lcm\") pod \"maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh\" (UID: \"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" Oct 31 13:45:00.481303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.481189 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead7008d-2414-412b-b024-2c4ddeda9399-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69\" (UID: \"ead7008d-2414-412b-b024-2c4ddeda9399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" Oct 31 13:45:00.481376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.481361 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkjs6\" (UniqueName: \"kubernetes.io/projected/c810f088-8712-4c24-bc79-1e19589746bb-kube-api-access-qkjs6\") pod \"maas-api-key-cleanup-29890905-5qpvj\" (UID: \"c810f088-8712-4c24-bc79-1e19589746bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" Oct 31 13:45:00.481450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.481436 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh\" (UID: \"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" Oct 31 13:45:00.481498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.481487 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw5pm\" (UniqueName: \"kubernetes.io/projected/b8b015f6-8071-4ff8-8269-801537622c10-kube-api-access-vw5pm\") pod \"maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5\" (UID: \"b8b015f6-8071-4ff8-8269-801537622c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" Oct 31 13:45:00.481548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.481533 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc6gq\" (UniqueName: \"kubernetes.io/projected/ead7008d-2414-412b-b024-2c4ddeda9399-kube-api-access-rc6gq\") pod \"maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69\" (UID: \"ead7008d-2414-412b-b024-2c4ddeda9399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" Oct 31 13:45:00.481673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.481658 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76565e81-762a-4d6d-b887-9c75556cbed3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp\" (UID: \"76565e81-762a-4d6d-b887-9c75556cbed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" Oct 31 13:45:00.481714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.481692 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gqg7\" (UniqueName: \"kubernetes.io/projected/62a4f033-5cb3-4d5e-998b-b66150794430-kube-api-access-6gqg7\") pod \"maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl\" (UID: \"62a4f033-5cb3-4d5e-998b-b66150794430\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" Oct 31 13:45:00.481777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.481756 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c810f088-8712-4c24-bc79-1e19589746bb-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890905-5qpvj\" (UID: \"c810f088-8712-4c24-bc79-1e19589746bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" Oct 31 13:45:00.481821 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.481761 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25tls\" (UniqueName: \"kubernetes.io/projected/76565e81-762a-4d6d-b887-9c75556cbed3-kube-api-access-25tls\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp\" (UID: \"76565e81-762a-4d6d-b887-9c75556cbed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" Oct 31 13:45:00.481923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.481901 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b015f6-8071-4ff8-8269-801537622c10-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5\" (UID: \"b8b015f6-8071-4ff8-8269-801537622c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" Oct 31 13:45:00.495329 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.495302 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw5pm\" (UniqueName: \"kubernetes.io/projected/b8b015f6-8071-4ff8-8269-801537622c10-kube-api-access-vw5pm\") pod \"maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5\" (UID: \"b8b015f6-8071-4ff8-8269-801537622c10\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" Oct 31 13:45:00.497710 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.497683 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkjs6\" (UniqueName: \"kubernetes.io/projected/c810f088-8712-4c24-bc79-1e19589746bb-kube-api-access-qkjs6\") pod \"maas-api-key-cleanup-29890905-5qpvj\" (UID: \"c810f088-8712-4c24-bc79-1e19589746bb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" Oct 31 13:45:00.544970 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.544936 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" Oct 31 13:45:00.568667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.563410 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" Oct 31 13:45:00.578544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.578512 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" Oct 31 13:45:00.585028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.584994 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76565e81-762a-4d6d-b887-9c75556cbed3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp\" (UID: \"76565e81-762a-4d6d-b887-9c75556cbed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" Oct 31 13:45:00.585148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.585044 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gqg7\" (UniqueName: \"kubernetes.io/projected/62a4f033-5cb3-4d5e-998b-b66150794430-kube-api-access-6gqg7\") pod \"maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl\" (UID: \"62a4f033-5cb3-4d5e-998b-b66150794430\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" Oct 31 13:45:00.585148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.585099 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25tls\" (UniqueName: \"kubernetes.io/projected/76565e81-762a-4d6d-b887-9c75556cbed3-kube-api-access-25tls\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp\" (UID: \"76565e81-762a-4d6d-b887-9c75556cbed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" Oct 31 13:45:00.585264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.585225 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a4f033-5cb3-4d5e-998b-b66150794430-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl\" (UID: \"62a4f033-5cb3-4d5e-998b-b66150794430\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" Oct 31 13:45:00.585322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.585259 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6lcm\" (UniqueName: \"kubernetes.io/projected/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-kube-api-access-t6lcm\") pod \"maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh\" (UID: \"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" Oct 31 13:45:00.585322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.585288 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead7008d-2414-412b-b024-2c4ddeda9399-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69\" (UID: \"ead7008d-2414-412b-b024-2c4ddeda9399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" Oct 31 13:45:00.585835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.585500 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh\" (UID: \"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" Oct 31 13:45:00.585835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.585566 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc6gq\" (UniqueName: \"kubernetes.io/projected/ead7008d-2414-412b-b024-2c4ddeda9399-kube-api-access-rc6gq\") pod \"maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69\" (UID: \"ead7008d-2414-412b-b024-2c4ddeda9399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" Oct 31 13:45:00.586134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.585921 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76565e81-762a-4d6d-b887-9c75556cbed3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp\" (UID: \"76565e81-762a-4d6d-b887-9c75556cbed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" Oct 31 13:45:00.587014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.586875 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead7008d-2414-412b-b024-2c4ddeda9399-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69\" (UID: \"ead7008d-2414-412b-b024-2c4ddeda9399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" Oct 31 13:45:00.587014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.586915 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a4f033-5cb3-4d5e-998b-b66150794430-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl\" (UID: \"62a4f033-5cb3-4d5e-998b-b66150794430\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" Oct 31 13:45:00.587014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.586965 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh\" (UID: \"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" Oct 31 13:45:00.599742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.599685 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25tls\" (UniqueName: \"kubernetes.io/projected/76565e81-762a-4d6d-b887-9c75556cbed3-kube-api-access-25tls\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp\" (UID: \"76565e81-762a-4d6d-b887-9c75556cbed3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" Oct 31 13:45:00.605308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.605278 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6lcm\" (UniqueName: \"kubernetes.io/projected/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-kube-api-access-t6lcm\") pod \"maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh\" (UID: \"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" Oct 31 13:45:00.605923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.605898 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gqg7\" (UniqueName: \"kubernetes.io/projected/62a4f033-5cb3-4d5e-998b-b66150794430-kube-api-access-6gqg7\") pod \"maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl\" (UID: \"62a4f033-5cb3-4d5e-998b-b66150794430\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" Oct 31 13:45:00.609082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.609053 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" Oct 31 13:45:00.616412 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.616383 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc6gq\" (UniqueName: \"kubernetes.io/projected/ead7008d-2414-412b-b024-2c4ddeda9399-kube-api-access-rc6gq\") pod \"maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69\" (UID: \"ead7008d-2414-412b-b024-2c4ddeda9399\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" Oct 31 13:45:00.640298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.640075 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" Oct 31 13:45:00.681025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.680984 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" Oct 31 13:45:00.687559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.687494 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" Oct 31 13:45:00.738067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.737891 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" Oct 31 13:45:00.748807 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.746565 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" Oct 31 13:45:00.794580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:00.794486 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6"] Oct 31 13:45:01.030771 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:45:01.030734 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba05975_872c_4c87_aff7_07080b68afb4.slice/crio-55005ae08df781207b79044f54d84122be9b592ad600076ba10729f0c377e347 WatchSource:0}: Error finding container 55005ae08df781207b79044f54d84122be9b592ad600076ba10729f0c377e347: Status 404 returned error can't find the container with id 55005ae08df781207b79044f54d84122be9b592ad600076ba10729f0c377e347 Oct 31 13:45:01.169858 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:01.169825 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" event={"ID":"aba05975-872c-4c87-aff7-07080b68afb4","Type":"ContainerStarted","Data":"55005ae08df781207b79044f54d84122be9b592ad600076ba10729f0c377e347"} Oct 31 13:45:01.273568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:01.273534 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m"] Oct 31 13:45:01.276950 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:45:01.276900 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd2e824_e299_438a_ae40_d3ebc513943d.slice/crio-3327adbd4198631ef91039ef0ae36c67856779286c6ec335942da56fec32804e WatchSource:0}: Error finding container 3327adbd4198631ef91039ef0ae36c67856779286c6ec335942da56fec32804e: Status 404 returned error can't find the container with id 3327adbd4198631ef91039ef0ae36c67856779286c6ec335942da56fec32804e Oct 31 13:45:01.365859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:01.365833 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z"] Oct 31 13:45:01.378747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:01.378713 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc"] Oct 31 13:45:01.388943 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:45:01.388898 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b015f6_8071_4ff8_8269_801537622c10.slice/crio-6f77e0a5964ad20a9c27cd0765d11579be90e4f2ccbc9cd92064c0f2d18ce0dc WatchSource:0}: Error finding container 6f77e0a5964ad20a9c27cd0765d11579be90e4f2ccbc9cd92064c0f2d18ce0dc: Status 404 returned error can't find the container with id 6f77e0a5964ad20a9c27cd0765d11579be90e4f2ccbc9cd92064c0f2d18ce0dc Oct 31 13:45:01.405486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:01.405443 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5"] Oct 31 13:45:01.574210 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:45:01.573042 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc810f088_8712_4c24_bc79_1e19589746bb.slice/crio-76483aab0adade8529bba1ccb30b6261ef7d8d141214844ae0f17354b361afe4 WatchSource:0}: Error finding container 76483aab0adade8529bba1ccb30b6261ef7d8d141214844ae0f17354b361afe4: Status 404 returned error can't find the container with id 76483aab0adade8529bba1ccb30b6261ef7d8d141214844ae0f17354b361afe4 Oct 31 13:45:01.580359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:01.580334 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj"] Oct 31 13:45:01.592268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:01.592195 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh"] Oct 31 13:45:01.602252 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:01.602220 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69"] Oct 31 13:45:01.625658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:01.625618 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp"] Oct 31 13:45:01.646716 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:45:01.646683 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76565e81_762a_4d6d_b887_9c75556cbed3.slice/crio-3ef1cc8aeabfc7ed8b9e149b76dfcd6b55e2a98c7704a564a66dfb0e099b4215 WatchSource:0}: Error finding container 3ef1cc8aeabfc7ed8b9e149b76dfcd6b55e2a98c7704a564a66dfb0e099b4215: Status 404 returned error can't find the container with id 3ef1cc8aeabfc7ed8b9e149b76dfcd6b55e2a98c7704a564a66dfb0e099b4215 Oct 31 13:45:01.652647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:01.652389 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl"] Oct 31 13:45:01.659728 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:45:01.659696 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a4f033_5cb3_4d5e_998b_b66150794430.slice/crio-8f5a26ac2ca5cc08ddb927d52737ae7a6ba71d540c72de327297abfe98b0a5da WatchSource:0}: Error finding container 8f5a26ac2ca5cc08ddb927d52737ae7a6ba71d540c72de327297abfe98b0a5da: Status 404 returned error can't find the container with id 8f5a26ac2ca5cc08ddb927d52737ae7a6ba71d540c72de327297abfe98b0a5da Oct 31 13:45:01.977049 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:01.977013 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:45:02.180299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:02.180255 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" event={"ID":"aba05975-872c-4c87-aff7-07080b68afb4","Type":"ContainerStarted","Data":"0bede5ce0faeedad0d394c5b7168f8d56f15e2a0d2aea9dbeb7981b16bbc43fe"} Oct 31 13:45:02.181748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:02.181724 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" event={"ID":"096af709-7898-469d-a55b-bbbacf1cb43e","Type":"ContainerStarted","Data":"5bc026922bbb6fc1bcab81aca476658b50ea46f289b1ce0cc15d1a05dc9b5d7f"} Oct 31 13:45:02.183054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:02.183024 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" event={"ID":"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b","Type":"ContainerStarted","Data":"85c20ae7f446da9e7c8ad40a6b1e932a6cf7e5a9db29664ef6d2ce867db83218"} Oct 31 13:45:02.184234 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:02.184211 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" event={"ID":"b8b015f6-8071-4ff8-8269-801537622c10","Type":"ContainerStarted","Data":"6f77e0a5964ad20a9c27cd0765d11579be90e4f2ccbc9cd92064c0f2d18ce0dc"} Oct 31 13:45:02.185401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:02.185372 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" event={"ID":"e97636dc-fff6-4b9d-adf8-8d29c8becb0d","Type":"ContainerStarted","Data":"2acb5f46b9d5d494cd4adb3fc8dd2cc36c6a70a1f9e8900cbdbcad85cd506d8c"} Oct 31 13:45:02.186856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:02.186835 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" event={"ID":"62a4f033-5cb3-4d5e-998b-b66150794430","Type":"ContainerStarted","Data":"8f5a26ac2ca5cc08ddb927d52737ae7a6ba71d540c72de327297abfe98b0a5da"} Oct 31 13:45:02.188067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:02.188040 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" event={"ID":"76565e81-762a-4d6d-b887-9c75556cbed3","Type":"ContainerStarted","Data":"3ef1cc8aeabfc7ed8b9e149b76dfcd6b55e2a98c7704a564a66dfb0e099b4215"} Oct 31 13:45:02.189518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:02.189498 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" event={"ID":"1cd2e824-e299-438a-ae40-d3ebc513943d","Type":"ContainerStarted","Data":"9d60d4bcc1fe8bbb56f9ae618d1b6a7f1e5743af03200de2b9d31dcdc9a18de2"} Oct 31 13:45:02.189633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:02.189545 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" event={"ID":"1cd2e824-e299-438a-ae40-d3ebc513943d","Type":"ContainerStarted","Data":"3327adbd4198631ef91039ef0ae36c67856779286c6ec335942da56fec32804e"} Oct 31 13:45:02.190854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:02.190831 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" event={"ID":"ead7008d-2414-412b-b024-2c4ddeda9399","Type":"ContainerStarted","Data":"5087f023bc7266d9a2295953e8068c3f51f0aa2595b1cd51241842795cfec166"} Oct 31 13:45:02.192068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:02.192049 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" event={"ID":"c810f088-8712-4c24-bc79-1e19589746bb","Type":"ContainerStarted","Data":"76483aab0adade8529bba1ccb30b6261ef7d8d141214844ae0f17354b361afe4"} Oct 31 13:45:02.206013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:02.205969 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" podStartSLOduration=2.205952865 podStartE2EDuration="2.205952865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:02.204306034 +0000 UTC m=+2470.350049861" watchObservedRunningTime="2026-10-31 13:45:02.205952865 +0000 UTC m=+2470.351696696" Oct 31 13:45:02.237903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:02.237850 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" podStartSLOduration=2.237836152 podStartE2EDuration="2.237836152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:02.234687635 +0000 UTC m=+2470.380431463" watchObservedRunningTime="2026-10-31 13:45:02.237836152 +0000 UTC m=+2470.383579977" Oct 31 13:45:03.203144 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.203109 5532 generic.go:356] "Generic (PLEG): container finished" podID="1cd2e824-e299-438a-ae40-d3ebc513943d" containerID="9d60d4bcc1fe8bbb56f9ae618d1b6a7f1e5743af03200de2b9d31dcdc9a18de2" exitCode=60 Oct 31 13:45:03.203536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.203199 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" event={"ID":"1cd2e824-e299-438a-ae40-d3ebc513943d","Type":"ContainerDied","Data":"9d60d4bcc1fe8bbb56f9ae618d1b6a7f1e5743af03200de2b9d31dcdc9a18de2"} Oct 31 13:45:03.203536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.203481 5532 scope.go:122] "RemoveContainer" containerID="9d60d4bcc1fe8bbb56f9ae618d1b6a7f1e5743af03200de2b9d31dcdc9a18de2" Oct 31 13:45:03.205131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.205109 5532 generic.go:356] "Generic (PLEG): container finished" podID="ead7008d-2414-412b-b024-2c4ddeda9399" containerID="6a19e4c80cfcc606767fffab438e63095de775e556ccad83a1068bf96cf7f1fa" exitCode=60 Oct 31 13:45:03.205239 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.205152 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" event={"ID":"ead7008d-2414-412b-b024-2c4ddeda9399","Type":"ContainerDied","Data":"6a19e4c80cfcc606767fffab438e63095de775e556ccad83a1068bf96cf7f1fa"} Oct 31 13:45:03.205553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.205447 5532 scope.go:122] "RemoveContainer" containerID="6a19e4c80cfcc606767fffab438e63095de775e556ccad83a1068bf96cf7f1fa" Oct 31 13:45:03.207382 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.207357 5532 generic.go:356] "Generic (PLEG): container finished" podID="c810f088-8712-4c24-bc79-1e19589746bb" containerID="4aebbf4d978bcb6ef3887e393a1578acf93b16624626f147a3348c140651c3a9" exitCode=60 Oct 31 13:45:03.207487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.207469 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" event={"ID":"c810f088-8712-4c24-bc79-1e19589746bb","Type":"ContainerDied","Data":"4aebbf4d978bcb6ef3887e393a1578acf93b16624626f147a3348c140651c3a9"} Oct 31 13:45:03.207755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.207739 5532 scope.go:122] "RemoveContainer" containerID="4aebbf4d978bcb6ef3887e393a1578acf93b16624626f147a3348c140651c3a9" Oct 31 13:45:03.209478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.209398 5532 generic.go:356] "Generic (PLEG): container finished" podID="aba05975-872c-4c87-aff7-07080b68afb4" containerID="0bede5ce0faeedad0d394c5b7168f8d56f15e2a0d2aea9dbeb7981b16bbc43fe" exitCode=60 Oct 31 13:45:03.209478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.209475 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" event={"ID":"aba05975-872c-4c87-aff7-07080b68afb4","Type":"ContainerDied","Data":"0bede5ce0faeedad0d394c5b7168f8d56f15e2a0d2aea9dbeb7981b16bbc43fe"} Oct 31 13:45:03.209977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.209956 5532 scope.go:122] "RemoveContainer" containerID="0bede5ce0faeedad0d394c5b7168f8d56f15e2a0d2aea9dbeb7981b16bbc43fe" Oct 31 13:45:03.217554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.217531 5532 generic.go:356] "Generic (PLEG): container finished" podID="096af709-7898-469d-a55b-bbbacf1cb43e" containerID="fb0d4f7303d757fd8091e86f5596868aa25336f6d120f0220ba9e0e8da20b287" exitCode=60 Oct 31 13:45:03.217686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.217643 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" event={"ID":"096af709-7898-469d-a55b-bbbacf1cb43e","Type":"ContainerDied","Data":"fb0d4f7303d757fd8091e86f5596868aa25336f6d120f0220ba9e0e8da20b287"} Oct 31 13:45:03.217876 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.217860 5532 scope.go:122] "RemoveContainer" containerID="fb0d4f7303d757fd8091e86f5596868aa25336f6d120f0220ba9e0e8da20b287" Oct 31 13:45:03.219798 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.219771 5532 generic.go:356] "Generic (PLEG): container finished" podID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" containerID="675aa6714f098bd8670f26dafa78813691a17c0539671cc729ba4974f885b117" exitCode=60 Oct 31 13:45:03.219886 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.219808 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" event={"ID":"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b","Type":"ContainerDied","Data":"675aa6714f098bd8670f26dafa78813691a17c0539671cc729ba4974f885b117"} Oct 31 13:45:03.220105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.220057 5532 scope.go:122] "RemoveContainer" containerID="675aa6714f098bd8670f26dafa78813691a17c0539671cc729ba4974f885b117" Oct 31 13:45:03.221629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.221584 5532 generic.go:356] "Generic (PLEG): container finished" podID="b8b015f6-8071-4ff8-8269-801537622c10" containerID="d3f35e9171d0b51146e58fc3328f603ef3e81f2ea33bb4d8825d2ac5bd4b6419" exitCode=60 Oct 31 13:45:03.221629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.221622 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" event={"ID":"b8b015f6-8071-4ff8-8269-801537622c10","Type":"ContainerDied","Data":"d3f35e9171d0b51146e58fc3328f603ef3e81f2ea33bb4d8825d2ac5bd4b6419"} Oct 31 13:45:03.222250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.222229 5532 scope.go:122] "RemoveContainer" containerID="d3f35e9171d0b51146e58fc3328f603ef3e81f2ea33bb4d8825d2ac5bd4b6419" Oct 31 13:45:03.226217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.226191 5532 generic.go:356] "Generic (PLEG): container finished" podID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" containerID="dc305e08e31ebdb59e4d40b6cc2824a0f3f1cf3bd944a5e7f4059a0e39e7f80b" exitCode=60 Oct 31 13:45:03.226325 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.226276 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" event={"ID":"e97636dc-fff6-4b9d-adf8-8d29c8becb0d","Type":"ContainerDied","Data":"dc305e08e31ebdb59e4d40b6cc2824a0f3f1cf3bd944a5e7f4059a0e39e7f80b"} Oct 31 13:45:03.226506 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.226489 5532 scope.go:122] "RemoveContainer" containerID="dc305e08e31ebdb59e4d40b6cc2824a0f3f1cf3bd944a5e7f4059a0e39e7f80b" Oct 31 13:45:03.228434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.228412 5532 generic.go:356] "Generic (PLEG): container finished" podID="62a4f033-5cb3-4d5e-998b-b66150794430" containerID="f7109088b2c712029d306de0e806efbc2043b7b810b313e6f0b1510de4dd76fd" exitCode=60 Oct 31 13:45:03.228528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.228494 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" event={"ID":"62a4f033-5cb3-4d5e-998b-b66150794430","Type":"ContainerDied","Data":"f7109088b2c712029d306de0e806efbc2043b7b810b313e6f0b1510de4dd76fd"} Oct 31 13:45:03.228774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.228756 5532 scope.go:122] "RemoveContainer" containerID="f7109088b2c712029d306de0e806efbc2043b7b810b313e6f0b1510de4dd76fd" Oct 31 13:45:03.230417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.230396 5532 generic.go:356] "Generic (PLEG): container finished" podID="76565e81-762a-4d6d-b887-9c75556cbed3" containerID="5313c8af57b6209be779f357c8856d1e968dfcabe374054f81913f4681d39fa7" exitCode=60 Oct 31 13:45:03.230498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.230441 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" event={"ID":"76565e81-762a-4d6d-b887-9c75556cbed3","Type":"ContainerDied","Data":"5313c8af57b6209be779f357c8856d1e968dfcabe374054f81913f4681d39fa7"} Oct 31 13:45:03.230693 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:03.230679 5532 scope.go:122] "RemoveContainer" containerID="5313c8af57b6209be779f357c8856d1e968dfcabe374054f81913f4681d39fa7" Oct 31 13:45:04.242007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.241893 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" event={"ID":"76565e81-762a-4d6d-b887-9c75556cbed3","Type":"ContainerStarted","Data":"9c7188f15c08a68e58fae1f328921788a761a0cc05a0f02dac53ab6a74dfc556"} Oct 31 13:45:04.245769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.245740 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" event={"ID":"1cd2e824-e299-438a-ae40-d3ebc513943d","Type":"ContainerStarted","Data":"c929e01ab64f42e729d01cd2bfce16e274b80fa935813910d4b69c62b03ebaef"} Oct 31 13:45:04.247999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.247975 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" event={"ID":"ead7008d-2414-412b-b024-2c4ddeda9399","Type":"ContainerStarted","Data":"3b8c15b2547b9b15ef231eaf133d8bdb92d6ef0da700c2ff6851761d08c73448"} Oct 31 13:45:04.250168 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.250142 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" event={"ID":"c810f088-8712-4c24-bc79-1e19589746bb","Type":"ContainerStarted","Data":"f637e5959a2fe689262693ea92317f150bd76add64cdda7a4f5668033280951f"} Oct 31 13:45:04.251994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.251968 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" event={"ID":"aba05975-872c-4c87-aff7-07080b68afb4","Type":"ContainerStarted","Data":"ce21e0704cb014c5e543ad1509bbc498b0b333e2314d519b1739d0d5b1c82a1d"} Oct 31 13:45:04.253987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.253960 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" event={"ID":"096af709-7898-469d-a55b-bbbacf1cb43e","Type":"ContainerStarted","Data":"9ed84e6597848cee04abf37261a5a3104d079e21f4bde952e7dac2362a46805e"} Oct 31 13:45:04.255943 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.255920 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" event={"ID":"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b","Type":"ContainerStarted","Data":"da3452be32b6e9605ab451ced0534196daa7f33a945fd22a54ecca811f3989df"} Oct 31 13:45:04.257800 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.257777 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" event={"ID":"b8b015f6-8071-4ff8-8269-801537622c10","Type":"ContainerStarted","Data":"d95cd10769b7da1dcc01e5c22bf2ff707935ea26d91b8f6c82f73cf5366f0034"} Oct 31 13:45:04.259817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.259795 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" event={"ID":"e97636dc-fff6-4b9d-adf8-8d29c8becb0d","Type":"ContainerStarted","Data":"a36c775932f5954b2a6223d4b914d2a9e143c2f1be25af95165f37c43db66b94"} Oct 31 13:45:04.261815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.261795 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" event={"ID":"62a4f033-5cb3-4d5e-998b-b66150794430","Type":"ContainerStarted","Data":"86d60185341f7e52c3815ea58e64ea3187f6aa0f25871c43c6d7f3bd6e7b8730"} Oct 31 13:45:04.271846 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.271804 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" podStartSLOduration=4.271790429 podStartE2EDuration="4.271790429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:04.267438249 +0000 UTC m=+2472.413182077" watchObservedRunningTime="2026-10-31 13:45:04.271790429 +0000 UTC m=+2472.417534257" Oct 31 13:45:04.292752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.292704 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" podStartSLOduration=4.292689756 podStartE2EDuration="4.292689756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:04.291307811 +0000 UTC m=+2472.437051642" watchObservedRunningTime="2026-10-31 13:45:04.292689756 +0000 UTC m=+2472.438433602" Oct 31 13:45:04.350162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.350106 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" podStartSLOduration=4.350089993 podStartE2EDuration="4.350089993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:04.346632076 +0000 UTC m=+2472.492375897" watchObservedRunningTime="2026-10-31 13:45:04.350089993 +0000 UTC m=+2472.495833820" Oct 31 13:45:04.369652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.369580 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" podStartSLOduration=4.369566342 podStartE2EDuration="4.369566342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:04.367130561 +0000 UTC m=+2472.512874390" watchObservedRunningTime="2026-10-31 13:45:04.369566342 +0000 UTC m=+2472.515310167" Oct 31 13:45:04.396356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.396301 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" podStartSLOduration=4.396282415 podStartE2EDuration="4.396282415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:04.388463889 +0000 UTC m=+2472.534221509" watchObservedRunningTime="2026-10-31 13:45:04.396282415 +0000 UTC m=+2472.542026246" Oct 31 13:45:04.415250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.415203 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" podStartSLOduration=4.415189915 podStartE2EDuration="4.415189915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:04.412551069 +0000 UTC m=+2472.558294898" watchObservedRunningTime="2026-10-31 13:45:04.415189915 +0000 UTC m=+2472.560933739" Oct 31 13:45:04.459885 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.459826 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" podStartSLOduration=4.459806779 podStartE2EDuration="4.459806779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:04.456729951 +0000 UTC m=+2472.602473779" watchObservedRunningTime="2026-10-31 13:45:04.459806779 +0000 UTC m=+2472.605550608" Oct 31 13:45:04.481428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:04.481382 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" podStartSLOduration=4.481369079 podStartE2EDuration="4.481369079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:04.476278795 +0000 UTC m=+2472.622022623" watchObservedRunningTime="2026-10-31 13:45:04.481369079 +0000 UTC m=+2472.627112907" Oct 31 13:45:05.272998 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.272964 5532 generic.go:356] "Generic (PLEG): container finished" podID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" containerID="da3452be32b6e9605ab451ced0534196daa7f33a945fd22a54ecca811f3989df" exitCode=60 Oct 31 13:45:05.273460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.273055 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" event={"ID":"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b","Type":"ContainerDied","Data":"da3452be32b6e9605ab451ced0534196daa7f33a945fd22a54ecca811f3989df"} Oct 31 13:45:05.273460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.273131 5532 scope.go:122] "RemoveContainer" containerID="675aa6714f098bd8670f26dafa78813691a17c0539671cc729ba4974f885b117" Oct 31 13:45:05.273460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.273413 5532 scope.go:122] "RemoveContainer" containerID="da3452be32b6e9605ab451ced0534196daa7f33a945fd22a54ecca811f3989df" Oct 31 13:45:05.273924 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:05.273902 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh_odh-ai-gateway-infra(4d93b407-6ae0-44b8-adab-96c1ed5b6d2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" Oct 31 13:45:05.275304 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.275278 5532 generic.go:356] "Generic (PLEG): container finished" podID="b8b015f6-8071-4ff8-8269-801537622c10" containerID="d95cd10769b7da1dcc01e5c22bf2ff707935ea26d91b8f6c82f73cf5366f0034" exitCode=60 Oct 31 13:45:05.275405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.275358 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" event={"ID":"b8b015f6-8071-4ff8-8269-801537622c10","Type":"ContainerDied","Data":"d95cd10769b7da1dcc01e5c22bf2ff707935ea26d91b8f6c82f73cf5366f0034"} Oct 31 13:45:05.275719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.275704 5532 scope.go:122] "RemoveContainer" containerID="d95cd10769b7da1dcc01e5c22bf2ff707935ea26d91b8f6c82f73cf5366f0034" Oct 31 13:45:05.276132 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:05.276113 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5_odh-ai-gateway-infra(b8b015f6-8071-4ff8-8269-801537622c10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" podUID="b8b015f6-8071-4ff8-8269-801537622c10" Oct 31 13:45:05.277293 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.277273 5532 generic.go:356] "Generic (PLEG): container finished" podID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" containerID="a36c775932f5954b2a6223d4b914d2a9e143c2f1be25af95165f37c43db66b94" exitCode=60 Oct 31 13:45:05.277377 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.277354 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" event={"ID":"e97636dc-fff6-4b9d-adf8-8d29c8becb0d","Type":"ContainerDied","Data":"a36c775932f5954b2a6223d4b914d2a9e143c2f1be25af95165f37c43db66b94"} Oct 31 13:45:05.277700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.277684 5532 scope.go:122] "RemoveContainer" containerID="a36c775932f5954b2a6223d4b914d2a9e143c2f1be25af95165f37c43db66b94" Oct 31 13:45:05.278098 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:05.278075 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z_odh-ai-gateway-infra(e97636dc-fff6-4b9d-adf8-8d29c8becb0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" podUID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" Oct 31 13:45:05.279521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.279499 5532 generic.go:356] "Generic (PLEG): container finished" podID="62a4f033-5cb3-4d5e-998b-b66150794430" containerID="86d60185341f7e52c3815ea58e64ea3187f6aa0f25871c43c6d7f3bd6e7b8730" exitCode=60 Oct 31 13:45:05.279680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.279565 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" event={"ID":"62a4f033-5cb3-4d5e-998b-b66150794430","Type":"ContainerDied","Data":"86d60185341f7e52c3815ea58e64ea3187f6aa0f25871c43c6d7f3bd6e7b8730"} Oct 31 13:45:05.279999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.279968 5532 scope.go:122] "RemoveContainer" containerID="86d60185341f7e52c3815ea58e64ea3187f6aa0f25871c43c6d7f3bd6e7b8730" Oct 31 13:45:05.280404 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:05.280386 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl_odh-ai-gateway-infra(62a4f033-5cb3-4d5e-998b-b66150794430)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" podUID="62a4f033-5cb3-4d5e-998b-b66150794430" Oct 31 13:45:05.282289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.282260 5532 generic.go:356] "Generic (PLEG): container finished" podID="76565e81-762a-4d6d-b887-9c75556cbed3" containerID="9c7188f15c08a68e58fae1f328921788a761a0cc05a0f02dac53ab6a74dfc556" exitCode=60 Oct 31 13:45:05.282385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.282346 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" event={"ID":"76565e81-762a-4d6d-b887-9c75556cbed3","Type":"ContainerDied","Data":"9c7188f15c08a68e58fae1f328921788a761a0cc05a0f02dac53ab6a74dfc556"} Oct 31 13:45:05.282629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.282609 5532 scope.go:122] "RemoveContainer" containerID="9c7188f15c08a68e58fae1f328921788a761a0cc05a0f02dac53ab6a74dfc556" Oct 31 13:45:05.283001 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:05.282978 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp_odh-ai-gateway-infra(76565e81-762a-4d6d-b887-9c75556cbed3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" podUID="76565e81-762a-4d6d-b887-9c75556cbed3" Oct 31 13:45:05.284429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.284408 5532 generic.go:356] "Generic (PLEG): container finished" podID="1cd2e824-e299-438a-ae40-d3ebc513943d" containerID="c929e01ab64f42e729d01cd2bfce16e274b80fa935813910d4b69c62b03ebaef" exitCode=60 Oct 31 13:45:05.284520 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.284491 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" event={"ID":"1cd2e824-e299-438a-ae40-d3ebc513943d","Type":"ContainerDied","Data":"c929e01ab64f42e729d01cd2bfce16e274b80fa935813910d4b69c62b03ebaef"} Oct 31 13:45:05.284794 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.284772 5532 scope.go:122] "RemoveContainer" containerID="c929e01ab64f42e729d01cd2bfce16e274b80fa935813910d4b69c62b03ebaef" Oct 31 13:45:05.285130 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:05.285110 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m_odh-ai-gateway-infra(1cd2e824-e299-438a-ae40-d3ebc513943d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" Oct 31 13:45:05.286654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.286585 5532 generic.go:356] "Generic (PLEG): container finished" podID="ead7008d-2414-412b-b024-2c4ddeda9399" containerID="3b8c15b2547b9b15ef231eaf133d8bdb92d6ef0da700c2ff6851761d08c73448" exitCode=60 Oct 31 13:45:05.286758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.286702 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" event={"ID":"ead7008d-2414-412b-b024-2c4ddeda9399","Type":"ContainerDied","Data":"3b8c15b2547b9b15ef231eaf133d8bdb92d6ef0da700c2ff6851761d08c73448"} Oct 31 13:45:05.286946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.286930 5532 scope.go:122] "RemoveContainer" containerID="3b8c15b2547b9b15ef231eaf133d8bdb92d6ef0da700c2ff6851761d08c73448" Oct 31 13:45:05.287304 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:05.287286 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69_odh-ai-gateway-infra(ead7008d-2414-412b-b024-2c4ddeda9399)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" Oct 31 13:45:05.288761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.288737 5532 generic.go:356] "Generic (PLEG): container finished" podID="c810f088-8712-4c24-bc79-1e19589746bb" containerID="f637e5959a2fe689262693ea92317f150bd76add64cdda7a4f5668033280951f" exitCode=60 Oct 31 13:45:05.288855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.288765 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" event={"ID":"c810f088-8712-4c24-bc79-1e19589746bb","Type":"ContainerDied","Data":"f637e5959a2fe689262693ea92317f150bd76add64cdda7a4f5668033280951f"} Oct 31 13:45:05.289027 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.289009 5532 scope.go:122] "RemoveContainer" containerID="f637e5959a2fe689262693ea92317f150bd76add64cdda7a4f5668033280951f" Oct 31 13:45:05.289348 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:05.289329 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890905-5qpvj_odh-ai-gateway-infra(c810f088-8712-4c24-bc79-1e19589746bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" podUID="c810f088-8712-4c24-bc79-1e19589746bb" Oct 31 13:45:05.290355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.290322 5532 scope.go:122] "RemoveContainer" containerID="d3f35e9171d0b51146e58fc3328f603ef3e81f2ea33bb4d8825d2ac5bd4b6419" Oct 31 13:45:05.291079 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.291057 5532 generic.go:356] "Generic (PLEG): container finished" podID="aba05975-872c-4c87-aff7-07080b68afb4" containerID="ce21e0704cb014c5e543ad1509bbc498b0b333e2314d519b1739d0d5b1c82a1d" exitCode=60 Oct 31 13:45:05.291178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.291138 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" event={"ID":"aba05975-872c-4c87-aff7-07080b68afb4","Type":"ContainerDied","Data":"ce21e0704cb014c5e543ad1509bbc498b0b333e2314d519b1739d0d5b1c82a1d"} Oct 31 13:45:05.291444 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.291424 5532 scope.go:122] "RemoveContainer" containerID="ce21e0704cb014c5e543ad1509bbc498b0b333e2314d519b1739d0d5b1c82a1d" Oct 31 13:45:05.291783 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:05.291761 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6_odh-ai-gateway-infra(aba05975-872c-4c87-aff7-07080b68afb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" podUID="aba05975-872c-4c87-aff7-07080b68afb4" Oct 31 13:45:05.293049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.293030 5532 generic.go:356] "Generic (PLEG): container finished" podID="096af709-7898-469d-a55b-bbbacf1cb43e" containerID="9ed84e6597848cee04abf37261a5a3104d079e21f4bde952e7dac2362a46805e" exitCode=60 Oct 31 13:45:05.293163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.293095 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" event={"ID":"096af709-7898-469d-a55b-bbbacf1cb43e","Type":"ContainerDied","Data":"9ed84e6597848cee04abf37261a5a3104d079e21f4bde952e7dac2362a46805e"} Oct 31 13:45:05.293347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.293334 5532 scope.go:122] "RemoveContainer" containerID="9ed84e6597848cee04abf37261a5a3104d079e21f4bde952e7dac2362a46805e" Oct 31 13:45:05.293580 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:05.293563 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc_odh-ai-gateway-infra(096af709-7898-469d-a55b-bbbacf1cb43e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" podUID="096af709-7898-469d-a55b-bbbacf1cb43e" Oct 31 13:45:05.307996 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.307978 5532 scope.go:122] "RemoveContainer" containerID="dc305e08e31ebdb59e4d40b6cc2824a0f3f1cf3bd944a5e7f4059a0e39e7f80b" Oct 31 13:45:05.325436 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.325412 5532 scope.go:122] "RemoveContainer" containerID="f7109088b2c712029d306de0e806efbc2043b7b810b313e6f0b1510de4dd76fd" Oct 31 13:45:05.341043 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.341022 5532 scope.go:122] "RemoveContainer" containerID="5313c8af57b6209be779f357c8856d1e968dfcabe374054f81913f4681d39fa7" Oct 31 13:45:05.356930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.356901 5532 scope.go:122] "RemoveContainer" containerID="9d60d4bcc1fe8bbb56f9ae618d1b6a7f1e5743af03200de2b9d31dcdc9a18de2" Oct 31 13:45:05.373026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.373000 5532 scope.go:122] "RemoveContainer" containerID="6a19e4c80cfcc606767fffab438e63095de775e556ccad83a1068bf96cf7f1fa" Oct 31 13:45:05.389211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.389193 5532 scope.go:122] "RemoveContainer" containerID="4aebbf4d978bcb6ef3887e393a1578acf93b16624626f147a3348c140651c3a9" Oct 31 13:45:05.415582 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.415553 5532 scope.go:122] "RemoveContainer" containerID="0bede5ce0faeedad0d394c5b7168f8d56f15e2a0d2aea9dbeb7981b16bbc43fe" Oct 31 13:45:05.435611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:05.435577 5532 scope.go:122] "RemoveContainer" containerID="fb0d4f7303d757fd8091e86f5596868aa25336f6d120f0220ba9e0e8da20b287" Oct 31 13:45:06.306479 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:06.306448 5532 scope.go:122] "RemoveContainer" containerID="da3452be32b6e9605ab451ced0534196daa7f33a945fd22a54ecca811f3989df" Oct 31 13:45:06.306998 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:06.306971 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh_odh-ai-gateway-infra(4d93b407-6ae0-44b8-adab-96c1ed5b6d2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" Oct 31 13:45:06.308380 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:06.308357 5532 scope.go:122] "RemoveContainer" containerID="d95cd10769b7da1dcc01e5c22bf2ff707935ea26d91b8f6c82f73cf5366f0034" Oct 31 13:45:06.308807 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:06.308780 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5_odh-ai-gateway-infra(b8b015f6-8071-4ff8-8269-801537622c10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" podUID="b8b015f6-8071-4ff8-8269-801537622c10" Oct 31 13:45:06.310196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:06.310173 5532 scope.go:122] "RemoveContainer" containerID="a36c775932f5954b2a6223d4b914d2a9e143c2f1be25af95165f37c43db66b94" Oct 31 13:45:06.310536 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:06.310507 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z_odh-ai-gateway-infra(e97636dc-fff6-4b9d-adf8-8d29c8becb0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" podUID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" Oct 31 13:45:06.312112 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:06.312093 5532 scope.go:122] "RemoveContainer" containerID="86d60185341f7e52c3815ea58e64ea3187f6aa0f25871c43c6d7f3bd6e7b8730" Oct 31 13:45:06.312434 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:06.312414 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl_odh-ai-gateway-infra(62a4f033-5cb3-4d5e-998b-b66150794430)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" podUID="62a4f033-5cb3-4d5e-998b-b66150794430" Oct 31 13:45:06.314131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:06.314112 5532 scope.go:122] "RemoveContainer" containerID="9c7188f15c08a68e58fae1f328921788a761a0cc05a0f02dac53ab6a74dfc556" Oct 31 13:45:06.314454 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:06.314428 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp_odh-ai-gateway-infra(76565e81-762a-4d6d-b887-9c75556cbed3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" podUID="76565e81-762a-4d6d-b887-9c75556cbed3" Oct 31 13:45:06.315842 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:06.315825 5532 scope.go:122] "RemoveContainer" containerID="c929e01ab64f42e729d01cd2bfce16e274b80fa935813910d4b69c62b03ebaef" Oct 31 13:45:06.316178 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:06.316158 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m_odh-ai-gateway-infra(1cd2e824-e299-438a-ae40-d3ebc513943d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" Oct 31 13:45:06.317455 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:06.317436 5532 scope.go:122] "RemoveContainer" containerID="3b8c15b2547b9b15ef231eaf133d8bdb92d6ef0da700c2ff6851761d08c73448" Oct 31 13:45:06.317782 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:06.317763 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69_odh-ai-gateway-infra(ead7008d-2414-412b-b024-2c4ddeda9399)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" Oct 31 13:45:06.319034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:06.319013 5532 scope.go:122] "RemoveContainer" containerID="f637e5959a2fe689262693ea92317f150bd76add64cdda7a4f5668033280951f" Oct 31 13:45:06.319339 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:06.319321 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890905-5qpvj_odh-ai-gateway-infra(c810f088-8712-4c24-bc79-1e19589746bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" podUID="c810f088-8712-4c24-bc79-1e19589746bb" Oct 31 13:45:06.320821 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:06.320802 5532 scope.go:122] "RemoveContainer" containerID="ce21e0704cb014c5e543ad1509bbc498b0b333e2314d519b1739d0d5b1c82a1d" Oct 31 13:45:06.321241 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:06.321206 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6_odh-ai-gateway-infra(aba05975-872c-4c87-aff7-07080b68afb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" podUID="aba05975-872c-4c87-aff7-07080b68afb4" Oct 31 13:45:06.322803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:06.322782 5532 scope.go:122] "RemoveContainer" containerID="9ed84e6597848cee04abf37261a5a3104d079e21f4bde952e7dac2362a46805e" Oct 31 13:45:06.323055 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:06.323033 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc_odh-ai-gateway-infra(096af709-7898-469d-a55b-bbbacf1cb43e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" podUID="096af709-7898-469d-a55b-bbbacf1cb43e" Oct 31 13:45:14.995863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:14.995830 5532 scope.go:122] "RemoveContainer" containerID="86d60185341f7e52c3815ea58e64ea3187f6aa0f25871c43c6d7f3bd6e7b8730" Oct 31 13:45:15.488333 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:15.488294 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" event={"ID":"62a4f033-5cb3-4d5e-998b-b66150794430","Type":"ContainerStarted","Data":"4987e6596576620b2da7e118351b8ce7bd1176fff26f2fc186daca708c978413"} Oct 31 13:45:15.997058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:15.997024 5532 scope.go:122] "RemoveContainer" containerID="a36c775932f5954b2a6223d4b914d2a9e143c2f1be25af95165f37c43db66b94" Oct 31 13:45:15.997549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:15.997246 5532 scope.go:122] "RemoveContainer" containerID="d95cd10769b7da1dcc01e5c22bf2ff707935ea26d91b8f6c82f73cf5366f0034" Oct 31 13:45:16.089964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:16.089936 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r"] Oct 31 13:45:16.090147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:16.090126 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" podUID="7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" containerName="istio-proxy" containerID="cri-o://009187c1a0f7e1005a9283a579bfe10caa6754c94ec8ccb43599fca073a4440b" gracePeriod=30 Oct 31 13:45:16.499549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:16.499508 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" event={"ID":"e97636dc-fff6-4b9d-adf8-8d29c8becb0d","Type":"ContainerStarted","Data":"e65194cbdff6654ceae9a249c27dfe65af26de311aa11667a27d2578db2f214e"} Oct 31 13:45:16.501552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:16.501529 5532 generic.go:356] "Generic (PLEG): container finished" podID="62a4f033-5cb3-4d5e-998b-b66150794430" containerID="4987e6596576620b2da7e118351b8ce7bd1176fff26f2fc186daca708c978413" exitCode=60 Oct 31 13:45:16.501780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:16.501636 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" event={"ID":"62a4f033-5cb3-4d5e-998b-b66150794430","Type":"ContainerDied","Data":"4987e6596576620b2da7e118351b8ce7bd1176fff26f2fc186daca708c978413"} Oct 31 13:45:16.501780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:16.501696 5532 scope.go:122] "RemoveContainer" containerID="86d60185341f7e52c3815ea58e64ea3187f6aa0f25871c43c6d7f3bd6e7b8730" Oct 31 13:45:16.502117 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:16.502102 5532 scope.go:122] "RemoveContainer" containerID="4987e6596576620b2da7e118351b8ce7bd1176fff26f2fc186daca708c978413" Oct 31 13:45:16.502538 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:16.502520 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl_odh-ai-gateway-infra(62a4f033-5cb3-4d5e-998b-b66150794430)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" podUID="62a4f033-5cb3-4d5e-998b-b66150794430" Oct 31 13:45:16.534043 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:16.534017 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl"] Oct 31 13:45:17.000620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.000313 5532 scope.go:122] "RemoveContainer" containerID="3b8c15b2547b9b15ef231eaf133d8bdb92d6ef0da700c2ff6851761d08c73448" Oct 31 13:45:17.011124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.001616 5532 scope.go:122] "RemoveContainer" containerID="c929e01ab64f42e729d01cd2bfce16e274b80fa935813910d4b69c62b03ebaef" Oct 31 13:45:17.040045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.040012 5532 scope.go:122] "RemoveContainer" containerID="f637e5959a2fe689262693ea92317f150bd76add64cdda7a4f5668033280951f" Oct 31 13:45:17.062633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.057358 5532 scope.go:122] "RemoveContainer" containerID="da3452be32b6e9605ab451ced0534196daa7f33a945fd22a54ecca811f3989df" Oct 31 13:45:17.522859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.522773 5532 generic.go:356] "Generic (PLEG): container finished" podID="b8b015f6-8071-4ff8-8269-801537622c10" containerID="74572a3b11ae3dfdb2c6b73aad8c77a7c769fbfb181079150edfd4b0532925e0" exitCode=60 Oct 31 13:45:17.523007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.522855 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" event={"ID":"b8b015f6-8071-4ff8-8269-801537622c10","Type":"ContainerDied","Data":"74572a3b11ae3dfdb2c6b73aad8c77a7c769fbfb181079150edfd4b0532925e0"} Oct 31 13:45:17.523007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.522924 5532 scope.go:122] "RemoveContainer" containerID="d95cd10769b7da1dcc01e5c22bf2ff707935ea26d91b8f6c82f73cf5366f0034" Oct 31 13:45:17.523478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.523457 5532 scope.go:122] "RemoveContainer" containerID="74572a3b11ae3dfdb2c6b73aad8c77a7c769fbfb181079150edfd4b0532925e0" Oct 31 13:45:17.524056 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:17.524030 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5_odh-ai-gateway-infra(b8b015f6-8071-4ff8-8269-801537622c10)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" podUID="b8b015f6-8071-4ff8-8269-801537622c10" Oct 31 13:45:17.528521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.528500 5532 generic.go:356] "Generic (PLEG): container finished" podID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" containerID="e65194cbdff6654ceae9a249c27dfe65af26de311aa11667a27d2578db2f214e" exitCode=60 Oct 31 13:45:17.528670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.528578 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" event={"ID":"e97636dc-fff6-4b9d-adf8-8d29c8becb0d","Type":"ContainerDied","Data":"e65194cbdff6654ceae9a249c27dfe65af26de311aa11667a27d2578db2f214e"} Oct 31 13:45:17.529039 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.529018 5532 scope.go:122] "RemoveContainer" containerID="e65194cbdff6654ceae9a249c27dfe65af26de311aa11667a27d2578db2f214e" Oct 31 13:45:17.529392 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:17.529370 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z_odh-ai-gateway-infra(e97636dc-fff6-4b9d-adf8-8d29c8becb0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" podUID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" Oct 31 13:45:17.542366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.542345 5532 scope.go:122] "RemoveContainer" containerID="a36c775932f5954b2a6223d4b914d2a9e143c2f1be25af95165f37c43db66b94" Oct 31 13:45:17.562640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.558239 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z"] Oct 31 13:45:17.755560 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.755532 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" Oct 31 13:45:17.770556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.770524 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62a4f033-5cb3-4d5e-998b-b66150794430-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a4f033-5cb3-4d5e-998b-b66150794430-maas-api-service-ca\") pod \"62a4f033-5cb3-4d5e-998b-b66150794430\" (UID: \"62a4f033-5cb3-4d5e-998b-b66150794430\") " Oct 31 13:45:17.770877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.770858 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62a4f033-5cb3-4d5e-998b-b66150794430-kube-api-access-6gqg7\" (UniqueName: \"kubernetes.io/projected/62a4f033-5cb3-4d5e-998b-b66150794430-kube-api-access-6gqg7\") pod \"62a4f033-5cb3-4d5e-998b-b66150794430\" (UID: \"62a4f033-5cb3-4d5e-998b-b66150794430\") " Oct 31 13:45:17.770971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.770939 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a4f033-5cb3-4d5e-998b-b66150794430-maas-api-service-ca" pod "62a4f033-5cb3-4d5e-998b-b66150794430" (UID: "62a4f033-5cb3-4d5e-998b-b66150794430"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:17.771569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.771547 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62a4f033-5cb3-4d5e-998b-b66150794430-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:17.842244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.842140 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a4f033-5cb3-4d5e-998b-b66150794430-kube-api-access-6gqg7" pod "62a4f033-5cb3-4d5e-998b-b66150794430" (UID: "62a4f033-5cb3-4d5e-998b-b66150794430"). InnerVolumeSpecName "kube-api-access-6gqg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:17.873066 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.873037 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gqg7\" (UniqueName: \"kubernetes.io/projected/62a4f033-5cb3-4d5e-998b-b66150794430-kube-api-access-6gqg7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:17.996254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:17.996215 5532 scope.go:122] "RemoveContainer" containerID="ce21e0704cb014c5e543ad1509bbc498b0b333e2314d519b1739d0d5b1c82a1d" Oct 31 13:45:18.556225 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.556202 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" Oct 31 13:45:18.556650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.556282 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl" event={"ID":"62a4f033-5cb3-4d5e-998b-b66150794430","Type":"ContainerDied","Data":"8f5a26ac2ca5cc08ddb927d52737ae7a6ba71d540c72de327297abfe98b0a5da"} Oct 31 13:45:18.556650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.556346 5532 scope.go:122] "RemoveContainer" containerID="4987e6596576620b2da7e118351b8ce7bd1176fff26f2fc186daca708c978413" Oct 31 13:45:18.558933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.558906 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" event={"ID":"1cd2e824-e299-438a-ae40-d3ebc513943d","Type":"ContainerStarted","Data":"8b90d8693382f7f11f94416b350364ac3fb3eefaecab3487df1e6f6c2194583a"} Oct 31 13:45:18.559609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.559426 5532 scope.go:122] "RemoveContainer" containerID="8b90d8693382f7f11f94416b350364ac3fb3eefaecab3487df1e6f6c2194583a" Oct 31 13:45:18.559856 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:18.559830 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m_odh-ai-gateway-infra(1cd2e824-e299-438a-ae40-d3ebc513943d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" Oct 31 13:45:18.561485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.561461 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" event={"ID":"ead7008d-2414-412b-b024-2c4ddeda9399","Type":"ContainerStarted","Data":"0b20dc40ec9853f41d57efe923ead835f1fa456155cc114cc955bad2280afdac"} Oct 31 13:45:18.561953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.561938 5532 scope.go:122] "RemoveContainer" containerID="0b20dc40ec9853f41d57efe923ead835f1fa456155cc114cc955bad2280afdac" Oct 31 13:45:18.562219 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:18.562201 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69_odh-ai-gateway-infra(ead7008d-2414-412b-b024-2c4ddeda9399)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" Oct 31 13:45:18.564498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.564477 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" event={"ID":"c810f088-8712-4c24-bc79-1e19589746bb","Type":"ContainerStarted","Data":"fd8d5f2c7ab8cbd30ca783bdb684348efc4c32831a8b6f747f829fab2b4f8677"} Oct 31 13:45:18.565034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.565016 5532 scope.go:122] "RemoveContainer" containerID="fd8d5f2c7ab8cbd30ca783bdb684348efc4c32831a8b6f747f829fab2b4f8677" Oct 31 13:45:18.565335 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:18.565312 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890905-5qpvj_odh-ai-gateway-infra(c810f088-8712-4c24-bc79-1e19589746bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" podUID="c810f088-8712-4c24-bc79-1e19589746bb" Oct 31 13:45:18.566739 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.566712 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" event={"ID":"aba05975-872c-4c87-aff7-07080b68afb4","Type":"ContainerStarted","Data":"32d2e026928b16eb7a99868160b0661638e9a331143a4b85dee296a13a7f53bc"} Oct 31 13:45:18.569983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.569956 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" event={"ID":"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b","Type":"ContainerStarted","Data":"581c9a9a6c13d3648511ea2264f1acb7e30b6b90d3b935c69e532349c6965ddb"} Oct 31 13:45:18.570417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.570397 5532 scope.go:122] "RemoveContainer" containerID="581c9a9a6c13d3648511ea2264f1acb7e30b6b90d3b935c69e532349c6965ddb" Oct 31 13:45:18.570750 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:18.570723 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh_odh-ai-gateway-infra(4d93b407-6ae0-44b8-adab-96c1ed5b6d2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" Oct 31 13:45:18.578240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.578219 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5"] Oct 31 13:45:18.837682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.835308 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" Oct 31 13:45:18.843025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.842997 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" Oct 31 13:45:18.849286 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.849262 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl"] Oct 31 13:45:18.854091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.854066 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-5286ea-29890905-hnxcl"] Oct 31 13:45:18.896357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.896324 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-maas-api-service-ca\") pod \"e97636dc-fff6-4b9d-adf8-8d29c8becb0d\" (UID: \"e97636dc-fff6-4b9d-adf8-8d29c8becb0d\") " Oct 31 13:45:18.896532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.896432 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b015f6-8071-4ff8-8269-801537622c10-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b015f6-8071-4ff8-8269-801537622c10-maas-api-service-ca\") pod \"b8b015f6-8071-4ff8-8269-801537622c10\" (UID: \"b8b015f6-8071-4ff8-8269-801537622c10\") " Oct 31 13:45:18.896650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.896559 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-kube-api-access-rp2v2\" (UniqueName: \"kubernetes.io/projected/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-kube-api-access-rp2v2\") pod \"e97636dc-fff6-4b9d-adf8-8d29c8becb0d\" (UID: \"e97636dc-fff6-4b9d-adf8-8d29c8becb0d\") " Oct 31 13:45:18.896756 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.896689 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b015f6-8071-4ff8-8269-801537622c10-kube-api-access-vw5pm\" (UniqueName: \"kubernetes.io/projected/b8b015f6-8071-4ff8-8269-801537622c10-kube-api-access-vw5pm\") pod \"b8b015f6-8071-4ff8-8269-801537622c10\" (UID: \"b8b015f6-8071-4ff8-8269-801537622c10\") " Oct 31 13:45:18.897728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.897667 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b015f6-8071-4ff8-8269-801537622c10-maas-api-service-ca" pod "b8b015f6-8071-4ff8-8269-801537622c10" (UID: "b8b015f6-8071-4ff8-8269-801537622c10"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:18.905176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.899158 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b015f6-8071-4ff8-8269-801537622c10-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:18.905176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.899688 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-maas-api-service-ca" pod "e97636dc-fff6-4b9d-adf8-8d29c8becb0d" (UID: "e97636dc-fff6-4b9d-adf8-8d29c8becb0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:18.908404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.908363 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b015f6-8071-4ff8-8269-801537622c10-kube-api-access-vw5pm" pod "b8b015f6-8071-4ff8-8269-801537622c10" (UID: "b8b015f6-8071-4ff8-8269-801537622c10"). InnerVolumeSpecName "kube-api-access-vw5pm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:18.909820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.909778 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-kube-api-access-rp2v2" pod "e97636dc-fff6-4b9d-adf8-8d29c8becb0d" (UID: "e97636dc-fff6-4b9d-adf8-8d29c8becb0d"). InnerVolumeSpecName "kube-api-access-rp2v2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:18.996418 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.996387 5532 scope.go:122] "RemoveContainer" containerID="9ed84e6597848cee04abf37261a5a3104d079e21f4bde952e7dac2362a46805e" Oct 31 13:45:18.996727 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:18.996708 5532 scope.go:122] "RemoveContainer" containerID="9c7188f15c08a68e58fae1f328921788a761a0cc05a0f02dac53ab6a74dfc556" Oct 31 13:45:19.009008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.003792 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp2v2\" (UniqueName: \"kubernetes.io/projected/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-kube-api-access-rp2v2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:19.009008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.003836 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw5pm\" (UniqueName: \"kubernetes.io/projected/b8b015f6-8071-4ff8-8269-801537622c10-kube-api-access-vw5pm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:19.009008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.003853 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e97636dc-fff6-4b9d-adf8-8d29c8becb0d-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:19.590510 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.590471 5532 generic.go:356] "Generic (PLEG): container finished" podID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" containerID="581c9a9a6c13d3648511ea2264f1acb7e30b6b90d3b935c69e532349c6965ddb" exitCode=60 Oct 31 13:45:19.590999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.590542 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" event={"ID":"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b","Type":"ContainerDied","Data":"581c9a9a6c13d3648511ea2264f1acb7e30b6b90d3b935c69e532349c6965ddb"} Oct 31 13:45:19.590999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.590657 5532 scope.go:122] "RemoveContainer" containerID="da3452be32b6e9605ab451ced0534196daa7f33a945fd22a54ecca811f3989df" Oct 31 13:45:19.591185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.591162 5532 scope.go:122] "RemoveContainer" containerID="581c9a9a6c13d3648511ea2264f1acb7e30b6b90d3b935c69e532349c6965ddb" Oct 31 13:45:19.591697 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:19.591676 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh_odh-ai-gateway-infra(4d93b407-6ae0-44b8-adab-96c1ed5b6d2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" Oct 31 13:45:19.592263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.592240 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" Oct 31 13:45:19.592394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.592262 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5" event={"ID":"b8b015f6-8071-4ff8-8269-801537622c10","Type":"ContainerDied","Data":"6f77e0a5964ad20a9c27cd0765d11579be90e4f2ccbc9cd92064c0f2d18ce0dc"} Oct 31 13:45:19.593847 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.593828 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" Oct 31 13:45:19.593936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.593851 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z" event={"ID":"e97636dc-fff6-4b9d-adf8-8d29c8becb0d","Type":"ContainerDied","Data":"2acb5f46b9d5d494cd4adb3fc8dd2cc36c6a70a1f9e8900cbdbcad85cd506d8c"} Oct 31 13:45:19.601273 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.601246 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" event={"ID":"76565e81-762a-4d6d-b887-9c75556cbed3","Type":"ContainerStarted","Data":"f01ca0019a7be9db1575f800bcc4268ad5a5d8f3d9fba27cd552c4faba3c78b1"} Oct 31 13:45:19.603773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.603750 5532 generic.go:356] "Generic (PLEG): container finished" podID="1cd2e824-e299-438a-ae40-d3ebc513943d" containerID="8b90d8693382f7f11f94416b350364ac3fb3eefaecab3487df1e6f6c2194583a" exitCode=60 Oct 31 13:45:19.603883 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.603818 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" event={"ID":"1cd2e824-e299-438a-ae40-d3ebc513943d","Type":"ContainerDied","Data":"8b90d8693382f7f11f94416b350364ac3fb3eefaecab3487df1e6f6c2194583a"} Oct 31 13:45:19.604222 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.604201 5532 scope.go:122] "RemoveContainer" containerID="8b90d8693382f7f11f94416b350364ac3fb3eefaecab3487df1e6f6c2194583a" Oct 31 13:45:19.604527 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:19.604502 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m_odh-ai-gateway-infra(1cd2e824-e299-438a-ae40-d3ebc513943d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" Oct 31 13:45:19.607466 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.607441 5532 generic.go:356] "Generic (PLEG): container finished" podID="ead7008d-2414-412b-b024-2c4ddeda9399" containerID="0b20dc40ec9853f41d57efe923ead835f1fa456155cc114cc955bad2280afdac" exitCode=60 Oct 31 13:45:19.607556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.607514 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" event={"ID":"ead7008d-2414-412b-b024-2c4ddeda9399","Type":"ContainerDied","Data":"0b20dc40ec9853f41d57efe923ead835f1fa456155cc114cc955bad2280afdac"} Oct 31 13:45:19.607908 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.607893 5532 scope.go:122] "RemoveContainer" containerID="74572a3b11ae3dfdb2c6b73aad8c77a7c769fbfb181079150edfd4b0532925e0" Oct 31 13:45:19.607978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.607947 5532 scope.go:122] "RemoveContainer" containerID="0b20dc40ec9853f41d57efe923ead835f1fa456155cc114cc955bad2280afdac" Oct 31 13:45:19.608330 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:19.608305 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69_odh-ai-gateway-infra(ead7008d-2414-412b-b024-2c4ddeda9399)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" Oct 31 13:45:19.610721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.610697 5532 generic.go:356] "Generic (PLEG): container finished" podID="c810f088-8712-4c24-bc79-1e19589746bb" containerID="fd8d5f2c7ab8cbd30ca783bdb684348efc4c32831a8b6f747f829fab2b4f8677" exitCode=60 Oct 31 13:45:19.610799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.610771 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" event={"ID":"c810f088-8712-4c24-bc79-1e19589746bb","Type":"ContainerDied","Data":"fd8d5f2c7ab8cbd30ca783bdb684348efc4c32831a8b6f747f829fab2b4f8677"} Oct 31 13:45:19.611196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.611175 5532 scope.go:122] "RemoveContainer" containerID="fd8d5f2c7ab8cbd30ca783bdb684348efc4c32831a8b6f747f829fab2b4f8677" Oct 31 13:45:19.611517 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:19.611499 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890905-5qpvj_odh-ai-gateway-infra(c810f088-8712-4c24-bc79-1e19589746bb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" podUID="c810f088-8712-4c24-bc79-1e19589746bb" Oct 31 13:45:19.614310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.614288 5532 generic.go:356] "Generic (PLEG): container finished" podID="aba05975-872c-4c87-aff7-07080b68afb4" containerID="32d2e026928b16eb7a99868160b0661638e9a331143a4b85dee296a13a7f53bc" exitCode=60 Oct 31 13:45:19.614854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.614423 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" event={"ID":"aba05975-872c-4c87-aff7-07080b68afb4","Type":"ContainerDied","Data":"32d2e026928b16eb7a99868160b0661638e9a331143a4b85dee296a13a7f53bc"} Oct 31 13:45:19.615364 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.615343 5532 scope.go:122] "RemoveContainer" containerID="32d2e026928b16eb7a99868160b0661638e9a331143a4b85dee296a13a7f53bc" Oct 31 13:45:19.616277 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.616253 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69"] Oct 31 13:45:19.618155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.618133 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" event={"ID":"096af709-7898-469d-a55b-bbbacf1cb43e","Type":"ContainerStarted","Data":"d1c122772e084c67a010d8f16b2f5c19462c35bd4dbf23f6db3f6dd79c9c36b9"} Oct 31 13:45:19.620947 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:19.620919 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6_odh-ai-gateway-infra(aba05975-872c-4c87-aff7-07080b68afb4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" podUID="aba05975-872c-4c87-aff7-07080b68afb4" Oct 31 13:45:19.623979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.623958 5532 scope.go:122] "RemoveContainer" containerID="e65194cbdff6654ceae9a249c27dfe65af26de311aa11667a27d2578db2f214e" Oct 31 13:45:19.635101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.635069 5532 scope.go:122] "RemoveContainer" containerID="c929e01ab64f42e729d01cd2bfce16e274b80fa935813910d4b69c62b03ebaef" Oct 31 13:45:19.646708 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.646684 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m"] Oct 31 13:45:19.654316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.654296 5532 scope.go:122] "RemoveContainer" containerID="3b8c15b2547b9b15ef231eaf133d8bdb92d6ef0da700c2ff6851761d08c73448" Oct 31 13:45:19.664232 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.664206 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj"] Oct 31 13:45:19.669523 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.669498 5532 scope.go:122] "RemoveContainer" containerID="f637e5959a2fe689262693ea92317f150bd76add64cdda7a4f5668033280951f" Oct 31 13:45:19.682222 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.682197 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh"] Oct 31 13:45:19.688704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.688676 5532 scope.go:122] "RemoveContainer" containerID="ce21e0704cb014c5e543ad1509bbc498b0b333e2314d519b1739d0d5b1c82a1d" Oct 31 13:45:19.717098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.717061 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6"] Oct 31 13:45:19.749404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.749376 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5"] Oct 31 13:45:19.762104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.762071 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-61da9d-29890905-4l4s5"] Oct 31 13:45:19.824416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.824376 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z"] Oct 31 13:45:19.831888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:19.831858 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-fb50fa-29890905-4d25z"] Oct 31 13:45:20.008461 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.008382 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62a4f033-5cb3-4d5e-998b-b66150794430" path="/var/lib/kubelet/pods/62a4f033-5cb3-4d5e-998b-b66150794430/volumes" Oct 31 13:45:20.008961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.008946 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b015f6-8071-4ff8-8269-801537622c10" path="/var/lib/kubelet/pods/b8b015f6-8071-4ff8-8269-801537622c10/volumes" Oct 31 13:45:20.009379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.009366 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" path="/var/lib/kubelet/pods/e97636dc-fff6-4b9d-adf8-8d29c8becb0d/volumes" Oct 31 13:45:20.632170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.632129 5532 generic.go:356] "Generic (PLEG): container finished" podID="76565e81-762a-4d6d-b887-9c75556cbed3" containerID="f01ca0019a7be9db1575f800bcc4268ad5a5d8f3d9fba27cd552c4faba3c78b1" exitCode=60 Oct 31 13:45:20.632762 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.632247 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" event={"ID":"76565e81-762a-4d6d-b887-9c75556cbed3","Type":"ContainerDied","Data":"f01ca0019a7be9db1575f800bcc4268ad5a5d8f3d9fba27cd552c4faba3c78b1"} Oct 31 13:45:20.632762 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.632313 5532 scope.go:122] "RemoveContainer" containerID="9c7188f15c08a68e58fae1f328921788a761a0cc05a0f02dac53ab6a74dfc556" Oct 31 13:45:20.632908 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.632889 5532 scope.go:122] "RemoveContainer" containerID="f01ca0019a7be9db1575f800bcc4268ad5a5d8f3d9fba27cd552c4faba3c78b1" Oct 31 13:45:20.633367 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:20.633341 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp_odh-ai-gateway-infra(76565e81-762a-4d6d-b887-9c75556cbed3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" podUID="76565e81-762a-4d6d-b887-9c75556cbed3" Oct 31 13:45:20.649572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.649539 5532 generic.go:356] "Generic (PLEG): container finished" podID="096af709-7898-469d-a55b-bbbacf1cb43e" containerID="d1c122772e084c67a010d8f16b2f5c19462c35bd4dbf23f6db3f6dd79c9c36b9" exitCode=60 Oct 31 13:45:20.649783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.649698 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" event={"ID":"096af709-7898-469d-a55b-bbbacf1cb43e","Type":"ContainerDied","Data":"d1c122772e084c67a010d8f16b2f5c19462c35bd4dbf23f6db3f6dd79c9c36b9"} Oct 31 13:45:20.650231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.650137 5532 scope.go:122] "RemoveContainer" containerID="d1c122772e084c67a010d8f16b2f5c19462c35bd4dbf23f6db3f6dd79c9c36b9" Oct 31 13:45:20.650777 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:20.650580 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc_odh-ai-gateway-infra(096af709-7898-469d-a55b-bbbacf1cb43e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" podUID="096af709-7898-469d-a55b-bbbacf1cb43e" Oct 31 13:45:20.657818 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.657798 5532 scope.go:122] "RemoveContainer" containerID="9ed84e6597848cee04abf37261a5a3104d079e21f4bde952e7dac2362a46805e" Oct 31 13:45:20.763780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.762688 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp"] Oct 31 13:45:20.823705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.823667 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w"] Oct 31 13:45:20.825211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825184 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" containerName="cleanup" Oct 31 13:45:20.825211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825211 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" containerName="cleanup" Oct 31 13:45:20.825409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825242 5532 cpu_manager.go:414] "containerMap: removing container" podUID="62a4f033-5cb3-4d5e-998b-b66150794430" containerName="cleanup" Oct 31 13:45:20.825409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825250 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a4f033-5cb3-4d5e-998b-b66150794430" containerName="cleanup" Oct 31 13:45:20.825409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825277 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b8b015f6-8071-4ff8-8269-801537622c10" containerName="cleanup" Oct 31 13:45:20.825409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825284 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b015f6-8071-4ff8-8269-801537622c10" containerName="cleanup" Oct 31 13:45:20.825409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825323 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" containerName="cleanup" Oct 31 13:45:20.825409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825331 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" containerName="cleanup" Oct 31 13:45:20.825409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825372 5532 cpu_manager.go:414] "containerMap: removing container" podUID="62a4f033-5cb3-4d5e-998b-b66150794430" containerName="cleanup" Oct 31 13:45:20.825409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825379 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a4f033-5cb3-4d5e-998b-b66150794430" containerName="cleanup" Oct 31 13:45:20.825409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825408 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b8b015f6-8071-4ff8-8269-801537622c10" containerName="cleanup" Oct 31 13:45:20.825900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825416 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b015f6-8071-4ff8-8269-801537622c10" containerName="cleanup" Oct 31 13:45:20.825900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825679 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b015f6-8071-4ff8-8269-801537622c10" containerName="cleanup" Oct 31 13:45:20.825900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825701 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b015f6-8071-4ff8-8269-801537622c10" containerName="cleanup" Oct 31 13:45:20.825900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825718 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" containerName="cleanup" Oct 31 13:45:20.825900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825730 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a4f033-5cb3-4d5e-998b-b66150794430" containerName="cleanup" Oct 31 13:45:20.825900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825742 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a4f033-5cb3-4d5e-998b-b66150794430" containerName="cleanup" Oct 31 13:45:20.825900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825757 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" containerName="cleanup" Oct 31 13:45:20.825900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825769 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" containerName="cleanup" Oct 31 13:45:20.825900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.825782 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="62a4f033-5cb3-4d5e-998b-b66150794430" containerName="cleanup" Oct 31 13:45:20.831803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.831778 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" Oct 31 13:45:20.836233 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.836208 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w"] Oct 31 13:45:20.839722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.839698 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:45:20.873201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.873175 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc"] Oct 31 13:45:20.964156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.941019 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" Oct 31 13:45:20.964156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.963287 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc49r\" (UniqueName: \"kubernetes.io/projected/b9c65a8d-5c95-4356-954d-106740ce3181-kube-api-access-tc49r\") pod \"maas-api-revoke-sub-67a40b26-r2n7w\" (UID: \"b9c65a8d-5c95-4356-954d-106740ce3181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" Oct 31 13:45:20.964156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.963430 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a40b26-r2n7w\" (UID: \"b9c65a8d-5c95-4356-954d-106740ce3181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" Oct 31 13:45:20.964156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:20.963525 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a40b26-r2n7w\" (UID: \"b9c65a8d-5c95-4356-954d-106740ce3181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" Oct 31 13:45:21.064917 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.064882 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ead7008d-2414-412b-b024-2c4ddeda9399-kube-api-access-rc6gq\" (UniqueName: \"kubernetes.io/projected/ead7008d-2414-412b-b024-2c4ddeda9399-kube-api-access-rc6gq\") pod \"ead7008d-2414-412b-b024-2c4ddeda9399\" (UID: \"ead7008d-2414-412b-b024-2c4ddeda9399\") " Oct 31 13:45:21.065261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.065237 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ead7008d-2414-412b-b024-2c4ddeda9399-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead7008d-2414-412b-b024-2c4ddeda9399-maas-api-service-ca\") pod \"ead7008d-2414-412b-b024-2c4ddeda9399\" (UID: \"ead7008d-2414-412b-b024-2c4ddeda9399\") " Oct 31 13:45:21.065556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.065533 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead7008d-2414-412b-b024-2c4ddeda9399-maas-api-service-ca" pod "ead7008d-2414-412b-b024-2c4ddeda9399" (UID: "ead7008d-2414-412b-b024-2c4ddeda9399"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:21.067724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.066062 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc49r\" (UniqueName: \"kubernetes.io/projected/b9c65a8d-5c95-4356-954d-106740ce3181-kube-api-access-tc49r\") pod \"maas-api-revoke-sub-67a40b26-r2n7w\" (UID: \"b9c65a8d-5c95-4356-954d-106740ce3181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" Oct 31 13:45:21.067724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.066479 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a40b26-r2n7w\" (UID: \"b9c65a8d-5c95-4356-954d-106740ce3181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" Oct 31 13:45:21.067724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.066805 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a40b26-r2n7w\" (UID: \"b9c65a8d-5c95-4356-954d-106740ce3181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" Oct 31 13:45:21.067724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.067270 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead7008d-2414-412b-b024-2c4ddeda9399-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.068168 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.068141 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67a40b26-r2n7w\" (UID: \"b9c65a8d-5c95-4356-954d-106740ce3181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" Oct 31 13:45:21.069527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.069503 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ead7008d-2414-412b-b024-2c4ddeda9399-kube-api-access-rc6gq" pod "ead7008d-2414-412b-b024-2c4ddeda9399" (UID: "ead7008d-2414-412b-b024-2c4ddeda9399"). InnerVolumeSpecName "kube-api-access-rc6gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:21.070095 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.070068 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67a40b26-r2n7w\" (UID: \"b9c65a8d-5c95-4356-954d-106740ce3181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" Oct 31 13:45:21.093084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.093057 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc49r\" (UniqueName: \"kubernetes.io/projected/b9c65a8d-5c95-4356-954d-106740ce3181-kube-api-access-tc49r\") pod \"maas-api-revoke-sub-67a40b26-r2n7w\" (UID: \"b9c65a8d-5c95-4356-954d-106740ce3181\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" Oct 31 13:45:21.108071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.108047 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" Oct 31 13:45:21.111273 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.111246 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" Oct 31 13:45:21.119862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.119780 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" Oct 31 13:45:21.157937 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.157908 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" Oct 31 13:45:21.168948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.168103 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-maas-api-service-ca\") pod \"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b\" (UID: \"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b\") " Oct 31 13:45:21.168948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.168252 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c810f088-8712-4c24-bc79-1e19589746bb-kube-api-access-qkjs6\" (UniqueName: \"kubernetes.io/projected/c810f088-8712-4c24-bc79-1e19589746bb-kube-api-access-qkjs6\") pod \"c810f088-8712-4c24-bc79-1e19589746bb\" (UID: \"c810f088-8712-4c24-bc79-1e19589746bb\") " Oct 31 13:45:21.168948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.168373 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c810f088-8712-4c24-bc79-1e19589746bb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c810f088-8712-4c24-bc79-1e19589746bb-maas-api-service-ca\") pod \"c810f088-8712-4c24-bc79-1e19589746bb\" (UID: \"c810f088-8712-4c24-bc79-1e19589746bb\") " Oct 31 13:45:21.168948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.168471 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba05975-872c-4c87-aff7-07080b68afb4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba05975-872c-4c87-aff7-07080b68afb4-maas-api-service-ca\") pod \"aba05975-872c-4c87-aff7-07080b68afb4\" (UID: \"aba05975-872c-4c87-aff7-07080b68afb4\") " Oct 31 13:45:21.168948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.168607 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-kube-api-access-t6lcm\" (UniqueName: \"kubernetes.io/projected/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-kube-api-access-t6lcm\") pod \"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b\" (UID: \"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b\") " Oct 31 13:45:21.168948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.168777 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba05975-872c-4c87-aff7-07080b68afb4-kube-api-access-85b42\" (UniqueName: \"kubernetes.io/projected/aba05975-872c-4c87-aff7-07080b68afb4-kube-api-access-85b42\") pod \"aba05975-872c-4c87-aff7-07080b68afb4\" (UID: \"aba05975-872c-4c87-aff7-07080b68afb4\") " Oct 31 13:45:21.171227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.169704 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6gq\" (UniqueName: \"kubernetes.io/projected/ead7008d-2414-412b-b024-2c4ddeda9399-kube-api-access-rc6gq\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.171227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.169789 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" Oct 31 13:45:21.171227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.170775 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c810f088-8712-4c24-bc79-1e19589746bb-maas-api-service-ca" pod "c810f088-8712-4c24-bc79-1e19589746bb" (UID: "c810f088-8712-4c24-bc79-1e19589746bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:21.171227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.171028 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-maas-api-service-ca" pod "4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" (UID: "4d93b407-6ae0-44b8-adab-96c1ed5b6d2b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:21.172058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.171980 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba05975-872c-4c87-aff7-07080b68afb4-maas-api-service-ca" pod "aba05975-872c-4c87-aff7-07080b68afb4" (UID: "aba05975-872c-4c87-aff7-07080b68afb4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:21.172493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.172437 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba05975-872c-4c87-aff7-07080b68afb4-kube-api-access-85b42" pod "aba05975-872c-4c87-aff7-07080b68afb4" (UID: "aba05975-872c-4c87-aff7-07080b68afb4"). InnerVolumeSpecName "kube-api-access-85b42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:21.173984 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.173918 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c810f088-8712-4c24-bc79-1e19589746bb-kube-api-access-qkjs6" pod "c810f088-8712-4c24-bc79-1e19589746bb" (UID: "c810f088-8712-4c24-bc79-1e19589746bb"). InnerVolumeSpecName "kube-api-access-qkjs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:21.177653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.174809 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-kube-api-access-t6lcm" pod "4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" (UID: "4d93b407-6ae0-44b8-adab-96c1ed5b6d2b"). InnerVolumeSpecName "kube-api-access-t6lcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:21.273642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.270718 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd2e824-e299-438a-ae40-d3ebc513943d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd2e824-e299-438a-ae40-d3ebc513943d-maas-api-service-ca\") pod \"1cd2e824-e299-438a-ae40-d3ebc513943d\" (UID: \"1cd2e824-e299-438a-ae40-d3ebc513943d\") " Oct 31 13:45:21.273642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.270808 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd2e824-e299-438a-ae40-d3ebc513943d-kube-api-access-qwv6l\" (UniqueName: \"kubernetes.io/projected/1cd2e824-e299-438a-ae40-d3ebc513943d-kube-api-access-qwv6l\") pod \"1cd2e824-e299-438a-ae40-d3ebc513943d\" (UID: \"1cd2e824-e299-438a-ae40-d3ebc513943d\") " Oct 31 13:45:21.273642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.271878 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd2e824-e299-438a-ae40-d3ebc513943d-maas-api-service-ca" pod "1cd2e824-e299-438a-ae40-d3ebc513943d" (UID: "1cd2e824-e299-438a-ae40-d3ebc513943d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:21.273642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.272002 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85b42\" (UniqueName: \"kubernetes.io/projected/aba05975-872c-4c87-aff7-07080b68afb4-kube-api-access-85b42\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.273642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.272028 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.273642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.272044 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkjs6\" (UniqueName: \"kubernetes.io/projected/c810f088-8712-4c24-bc79-1e19589746bb-kube-api-access-qkjs6\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.273642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.272062 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c810f088-8712-4c24-bc79-1e19589746bb-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.273642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.272076 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba05975-872c-4c87-aff7-07080b68afb4-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.273642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.272090 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6lcm\" (UniqueName: \"kubernetes.io/projected/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b-kube-api-access-t6lcm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.284701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.284671 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd2e824-e299-438a-ae40-d3ebc513943d-kube-api-access-qwv6l" pod "1cd2e824-e299-438a-ae40-d3ebc513943d" (UID: "1cd2e824-e299-438a-ae40-d3ebc513943d"). InnerVolumeSpecName "kube-api-access-qwv6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:21.299890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.299854 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf"] Oct 31 13:45:21.301335 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301306 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" containerName="cleanup" Oct 31 13:45:21.301335 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301332 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301348 5532 cpu_manager.go:414] "containerMap: removing container" podUID="62a4f033-5cb3-4d5e-998b-b66150794430" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301356 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62a4f033-5cb3-4d5e-998b-b66150794430" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301372 5532 cpu_manager.go:414] "containerMap: removing container" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301380 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301394 5532 cpu_manager.go:414] "containerMap: removing container" podUID="aba05975-872c-4c87-aff7-07080b68afb4" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301401 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba05975-872c-4c87-aff7-07080b68afb4" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301418 5532 cpu_manager.go:414] "containerMap: removing container" podUID="aba05975-872c-4c87-aff7-07080b68afb4" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301425 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba05975-872c-4c87-aff7-07080b68afb4" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301439 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c810f088-8712-4c24-bc79-1e19589746bb" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301446 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c810f088-8712-4c24-bc79-1e19589746bb" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301456 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301464 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301477 5532 cpu_manager.go:414] "containerMap: removing container" podUID="aba05975-872c-4c87-aff7-07080b68afb4" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301484 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba05975-872c-4c87-aff7-07080b68afb4" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301499 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301508 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301537 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" containerName="cleanup" Oct 31 13:45:21.301534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301544 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e97636dc-fff6-4b9d-adf8-8d29c8becb0d" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301615 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c810f088-8712-4c24-bc79-1e19589746bb" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301624 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c810f088-8712-4c24-bc79-1e19589746bb" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301642 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b8b015f6-8071-4ff8-8269-801537622c10" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301649 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b015f6-8071-4ff8-8269-801537622c10" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301669 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301675 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301690 5532 cpu_manager.go:414] "containerMap: removing container" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301697 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.301956 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.302027 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.302043 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.302058 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.302066 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba05975-872c-4c87-aff7-07080b68afb4" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.302078 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba05975-872c-4c87-aff7-07080b68afb4" containerName="cleanup" Oct 31 13:45:21.302716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.302087 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c810f088-8712-4c24-bc79-1e19589746bb" containerName="cleanup" Oct 31 13:45:21.304682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.304661 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" containerName="cleanup" Oct 31 13:45:21.304826 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.304813 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba05975-872c-4c87-aff7-07080b68afb4" containerName="cleanup" Oct 31 13:45:21.305004 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.304936 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c810f088-8712-4c24-bc79-1e19589746bb" containerName="cleanup" Oct 31 13:45:21.305127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.305116 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b015f6-8071-4ff8-8269-801537622c10" containerName="cleanup" Oct 31 13:45:21.305211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.305201 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" containerName="cleanup" Oct 31 13:45:21.311042 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.311025 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf"] Oct 31 13:45:21.311185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.311166 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" Oct 31 13:45:21.391808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.391754 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf\" (UID: \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" Oct 31 13:45:21.392136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.392110 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf\" (UID: \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" Oct 31 13:45:21.392613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.392575 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm97r\" (UniqueName: \"kubernetes.io/projected/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-kube-api-access-jm97r\") pod \"maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf\" (UID: \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" Oct 31 13:45:21.392986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.392968 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1cd2e824-e299-438a-ae40-d3ebc513943d-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.393089 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.393077 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwv6l\" (UniqueName: \"kubernetes.io/projected/1cd2e824-e299-438a-ae40-d3ebc513943d-kube-api-access-qwv6l\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.496635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.496560 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf\" (UID: \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" Oct 31 13:45:21.496928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.496909 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf\" (UID: \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" Oct 31 13:45:21.502535 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.502498 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf\" (UID: \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" Oct 31 13:45:21.503403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.503370 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm97r\" (UniqueName: \"kubernetes.io/projected/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-kube-api-access-jm97r\") pod \"maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf\" (UID: \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" Oct 31 13:45:21.508064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.508008 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf\" (UID: \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" Oct 31 13:45:21.527640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.527425 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w"] Oct 31 13:45:21.559841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.544352 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm97r\" (UniqueName: \"kubernetes.io/projected/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-kube-api-access-jm97r\") pod \"maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf\" (UID: \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" Oct 31 13:45:21.623425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.623393 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:45:21.630460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.630439 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" Oct 31 13:45:21.671200 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.671168 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" Oct 31 13:45:21.671962 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.671185 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m" event={"ID":"1cd2e824-e299-438a-ae40-d3ebc513943d","Type":"ContainerDied","Data":"3327adbd4198631ef91039ef0ae36c67856779286c6ec335942da56fec32804e"} Oct 31 13:45:21.672097 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.672019 5532 scope.go:122] "RemoveContainer" containerID="8b90d8693382f7f11f94416b350364ac3fb3eefaecab3487df1e6f6c2194583a" Oct 31 13:45:21.673679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.673643 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" event={"ID":"ead7008d-2414-412b-b024-2c4ddeda9399","Type":"ContainerDied","Data":"5087f023bc7266d9a2295953e8068c3f51f0aa2595b1cd51241842795cfec166"} Oct 31 13:45:21.676243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.673797 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69" Oct 31 13:45:21.678006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.677982 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" event={"ID":"b9c65a8d-5c95-4356-954d-106740ce3181","Type":"ContainerStarted","Data":"2fcb2d810411361d04e2e4842999ddbf9350ea1e6db2f8e6d5a941668b1a76a6"} Oct 31 13:45:21.679983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.679961 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" event={"ID":"c810f088-8712-4c24-bc79-1e19589746bb","Type":"ContainerDied","Data":"76483aab0adade8529bba1ccb30b6261ef7d8d141214844ae0f17354b361afe4"} Oct 31 13:45:21.680195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.680181 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj" Oct 31 13:45:21.683242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.682475 5532 generic.go:356] "Generic (PLEG): container finished" podID="7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" containerID="009187c1a0f7e1005a9283a579bfe10caa6754c94ec8ccb43599fca073a4440b" exitCode=0 Oct 31 13:45:21.683242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.682552 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" event={"ID":"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98","Type":"ContainerDied","Data":"009187c1a0f7e1005a9283a579bfe10caa6754c94ec8ccb43599fca073a4440b"} Oct 31 13:45:21.683242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.682605 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" event={"ID":"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98","Type":"ContainerDied","Data":"1fa4ce02200a5dbea91d586ee35625810d89dd2c51cfbee324d30e620c0661d8"} Oct 31 13:45:21.683242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.682682 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r" Oct 31 13:45:21.697074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.697048 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" event={"ID":"aba05975-872c-4c87-aff7-07080b68afb4","Type":"ContainerDied","Data":"55005ae08df781207b79044f54d84122be9b592ad600076ba10729f0c377e347"} Oct 31 13:45:21.697074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.697072 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6" Oct 31 13:45:21.697542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.697524 5532 scope.go:122] "RemoveContainer" containerID="0b20dc40ec9853f41d57efe923ead835f1fa456155cc114cc955bad2280afdac" Oct 31 13:45:21.701663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.701637 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" event={"ID":"4d93b407-6ae0-44b8-adab-96c1ed5b6d2b","Type":"ContainerDied","Data":"85c20ae7f446da9e7c8ad40a6b1e932a6cf7e5a9db29664ef6d2ce867db83218"} Oct 31 13:45:21.701748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.701708 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh" Oct 31 13:45:21.710963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.710940 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-certs\") pod \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " Oct 31 13:45:21.711161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.711145 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-token\" (UniqueName: \"kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-token\") pod \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " Oct 31 13:45:21.711217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.711202 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-credential-socket\") pod \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " Oct 31 13:45:21.711275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.711235 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-certs" pod "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" (UID: "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:21.711275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.711255 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-data\") pod \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " Oct 31 13:45:21.711377 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.711344 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-socket\") pod \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " Oct 31 13:45:21.711450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.711433 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-envoy\") pod \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " Oct 31 13:45:21.711780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.711539 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istiod-ca-cert\") pod \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " Oct 31 13:45:21.711780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.711578 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-kube-api-access-gz7m7\" (UniqueName: \"kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-kube-api-access-gz7m7\") pod \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " Oct 31 13:45:21.711780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.711642 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-podinfo\") pod \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\" (UID: \"7f805bdf-0bd7-4ea9-8378-18f10dbc7d98\") " Oct 31 13:45:21.712274 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.712252 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-socket" pod "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" (UID: "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:21.712671 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.712651 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-credential-socket" pod "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" (UID: "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:21.713035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.713016 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-data" pod "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" (UID: "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:21.713238 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.713158 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istiod-ca-cert" pod "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" (UID: "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:21.715209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.715182 5532 scope.go:122] "RemoveContainer" containerID="fd8d5f2c7ab8cbd30ca783bdb684348efc4c32831a8b6f747f829fab2b4f8677" Oct 31 13:45:21.715302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.715272 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.715379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.715316 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.715379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.715332 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.715379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.715346 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.742757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.742377 5532 scope.go:122] "RemoveContainer" containerID="009187c1a0f7e1005a9283a579bfe10caa6754c94ec8ccb43599fca073a4440b" Oct 31 13:45:21.746238 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.746213 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-kube-api-access-gz7m7" pod "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" (UID: "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98"). InnerVolumeSpecName "kube-api-access-gz7m7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:21.746578 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.746532 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-token" pod "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" (UID: "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:21.746712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.746688 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-envoy" pod "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" (UID: "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:21.747193 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.747165 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-podinfo" pod "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" (UID: "7f805bdf-0bd7-4ea9-8378-18f10dbc7d98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:45:21.821821 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.821795 5532 scope.go:122] "RemoveContainer" containerID="009187c1a0f7e1005a9283a579bfe10caa6754c94ec8ccb43599fca073a4440b" Oct 31 13:45:21.822419 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:21.822382 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"009187c1a0f7e1005a9283a579bfe10caa6754c94ec8ccb43599fca073a4440b\": container with ID starting with 009187c1a0f7e1005a9283a579bfe10caa6754c94ec8ccb43599fca073a4440b not found: ID does not exist" containerID="009187c1a0f7e1005a9283a579bfe10caa6754c94ec8ccb43599fca073a4440b" Oct 31 13:45:21.822519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.822433 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"009187c1a0f7e1005a9283a579bfe10caa6754c94ec8ccb43599fca073a4440b"} err="failed to get container status \"009187c1a0f7e1005a9283a579bfe10caa6754c94ec8ccb43599fca073a4440b\": rpc error: code = NotFound desc = could not find container \"009187c1a0f7e1005a9283a579bfe10caa6754c94ec8ccb43599fca073a4440b\": container with ID starting with 009187c1a0f7e1005a9283a579bfe10caa6754c94ec8ccb43599fca073a4440b not found: ID does not exist" Oct 31 13:45:21.822519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.822460 5532 scope.go:122] "RemoveContainer" containerID="32d2e026928b16eb7a99868160b0661638e9a331143a4b85dee296a13a7f53bc" Oct 31 13:45:21.825516 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.825484 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.825680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.825665 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.825833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.825817 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.826180 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.826162 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.826311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.826296 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz7m7\" (UniqueName: \"kubernetes.io/projected/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98-kube-api-access-gz7m7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:21.831567 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.831544 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6"] Oct 31 13:45:21.839139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.839079 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-62778a-29890905-dnqm6"] Oct 31 13:45:21.846335 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.845476 5532 scope.go:122] "RemoveContainer" containerID="581c9a9a6c13d3648511ea2264f1acb7e30b6b90d3b935c69e532349c6965ddb" Oct 31 13:45:21.865814 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.865272 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69"] Oct 31 13:45:21.873347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.873311 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-72c210-29890905-s9l69"] Oct 31 13:45:21.879421 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.879391 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m"] Oct 31 13:45:21.891651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.888684 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-b06b08-29890905-6tf2m"] Oct 31 13:45:21.925375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.924701 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj"] Oct 31 13:45:21.936659 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.936588 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890905-5qpvj"] Oct 31 13:45:21.943684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.943655 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh"] Oct 31 13:45:21.950906 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:21.950882 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-4fde10-29890905-9qzvh"] Oct 31 13:45:22.019718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.019582 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" path="/var/lib/kubelet/pods/1cd2e824-e299-438a-ae40-d3ebc513943d/volumes" Oct 31 13:45:22.020371 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.020348 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" path="/var/lib/kubelet/pods/4d93b407-6ae0-44b8-adab-96c1ed5b6d2b/volumes" Oct 31 13:45:22.027177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.027145 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba05975-872c-4c87-aff7-07080b68afb4" path="/var/lib/kubelet/pods/aba05975-872c-4c87-aff7-07080b68afb4/volumes" Oct 31 13:45:22.027936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.027915 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c810f088-8712-4c24-bc79-1e19589746bb" path="/var/lib/kubelet/pods/c810f088-8712-4c24-bc79-1e19589746bb/volumes" Oct 31 13:45:22.028798 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.028741 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" path="/var/lib/kubelet/pods/ead7008d-2414-412b-b024-2c4ddeda9399/volumes" Oct 31 13:45:22.039573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.039548 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r"] Oct 31 13:45:22.048029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.048008 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-c4h9r"] Oct 31 13:45:22.081714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.081683 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf"] Oct 31 13:45:22.207248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.207222 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" Oct 31 13:45:22.210478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.210455 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" Oct 31 13:45:22.343082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.342996 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/096af709-7898-469d-a55b-bbbacf1cb43e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096af709-7898-469d-a55b-bbbacf1cb43e-maas-api-service-ca\") pod \"096af709-7898-469d-a55b-bbbacf1cb43e\" (UID: \"096af709-7898-469d-a55b-bbbacf1cb43e\") " Oct 31 13:45:22.343228 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.343209 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76565e81-762a-4d6d-b887-9c75556cbed3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76565e81-762a-4d6d-b887-9c75556cbed3-maas-api-service-ca\") pod \"76565e81-762a-4d6d-b887-9c75556cbed3\" (UID: \"76565e81-762a-4d6d-b887-9c75556cbed3\") " Oct 31 13:45:22.343415 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.343393 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/096af709-7898-469d-a55b-bbbacf1cb43e-maas-api-service-ca" pod "096af709-7898-469d-a55b-bbbacf1cb43e" (UID: "096af709-7898-469d-a55b-bbbacf1cb43e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:22.343553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.343536 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76565e81-762a-4d6d-b887-9c75556cbed3-kube-api-access-25tls\" (UniqueName: \"kubernetes.io/projected/76565e81-762a-4d6d-b887-9c75556cbed3-kube-api-access-25tls\") pod \"76565e81-762a-4d6d-b887-9c75556cbed3\" (UID: \"76565e81-762a-4d6d-b887-9c75556cbed3\") " Oct 31 13:45:22.343715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.343693 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/096af709-7898-469d-a55b-bbbacf1cb43e-kube-api-access-qhpz2\" (UniqueName: \"kubernetes.io/projected/096af709-7898-469d-a55b-bbbacf1cb43e-kube-api-access-qhpz2\") pod \"096af709-7898-469d-a55b-bbbacf1cb43e\" (UID: \"096af709-7898-469d-a55b-bbbacf1cb43e\") " Oct 31 13:45:22.343864 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.343834 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76565e81-762a-4d6d-b887-9c75556cbed3-maas-api-service-ca" pod "76565e81-762a-4d6d-b887-9c75556cbed3" (UID: "76565e81-762a-4d6d-b887-9c75556cbed3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:22.344528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.344504 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/096af709-7898-469d-a55b-bbbacf1cb43e-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:22.344528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.344528 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/76565e81-762a-4d6d-b887-9c75556cbed3-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:22.345876 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.345852 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76565e81-762a-4d6d-b887-9c75556cbed3-kube-api-access-25tls" pod "76565e81-762a-4d6d-b887-9c75556cbed3" (UID: "76565e81-762a-4d6d-b887-9c75556cbed3"). InnerVolumeSpecName "kube-api-access-25tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:22.345966 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.345891 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/096af709-7898-469d-a55b-bbbacf1cb43e-kube-api-access-qhpz2" pod "096af709-7898-469d-a55b-bbbacf1cb43e" (UID: "096af709-7898-469d-a55b-bbbacf1cb43e"). InnerVolumeSpecName "kube-api-access-qhpz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:22.446418 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.446383 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25tls\" (UniqueName: \"kubernetes.io/projected/76565e81-762a-4d6d-b887-9c75556cbed3-kube-api-access-25tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:22.446418 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.446419 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhpz2\" (UniqueName: \"kubernetes.io/projected/096af709-7898-469d-a55b-bbbacf1cb43e-kube-api-access-qhpz2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:22.728974 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.728945 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" Oct 31 13:45:22.729466 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.728980 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc" event={"ID":"096af709-7898-469d-a55b-bbbacf1cb43e","Type":"ContainerDied","Data":"5bc026922bbb6fc1bcab81aca476658b50ea46f289b1ce0cc15d1a05dc9b5d7f"} Oct 31 13:45:22.729466 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.729048 5532 scope.go:122] "RemoveContainer" containerID="d1c122772e084c67a010d8f16b2f5c19462c35bd4dbf23f6db3f6dd79c9c36b9" Oct 31 13:45:22.732217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.732186 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" event={"ID":"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8","Type":"ContainerStarted","Data":"3445af2d9cff82b521d3fafaa79bcb56e3f67233c1b7f756e497858252fdd6a2"} Oct 31 13:45:22.732364 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.732237 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" event={"ID":"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8","Type":"ContainerStarted","Data":"317f105681d0b543edce0c0b057a192630924950eb295111124645a4001386cf"} Oct 31 13:45:22.735059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.735023 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" Oct 31 13:45:22.735059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.735038 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp" event={"ID":"76565e81-762a-4d6d-b887-9c75556cbed3","Type":"ContainerDied","Data":"3ef1cc8aeabfc7ed8b9e149b76dfcd6b55e2a98c7704a564a66dfb0e099b4215"} Oct 31 13:45:22.739287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.739259 5532 generic.go:356] "Generic (PLEG): container finished" podID="b9c65a8d-5c95-4356-954d-106740ce3181" containerID="041de7b4d31e558b71225af9e41ab82253c1afb87bd36d8314063aaee1c65c4b" exitCode=0 Oct 31 13:45:22.739438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.739346 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" event={"ID":"b9c65a8d-5c95-4356-954d-106740ce3181","Type":"ContainerDied","Data":"041de7b4d31e558b71225af9e41ab82253c1afb87bd36d8314063aaee1c65c4b"} Oct 31 13:45:22.744951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.744930 5532 scope.go:122] "RemoveContainer" containerID="f01ca0019a7be9db1575f800bcc4268ad5a5d8f3d9fba27cd552c4faba3c78b1" Oct 31 13:45:22.750418 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:22.750384 5532 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9c65a8d_5c95_4356_954d_106740ce3181.slice/crio-conmon-041de7b4d31e558b71225af9e41ab82253c1afb87bd36d8314063aaee1c65c4b.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:45:22.784920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.784875 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" podStartSLOduration=1.784861273 podStartE2EDuration="1.784861273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:22.778315571 +0000 UTC m=+2490.924059400" watchObservedRunningTime="2026-10-31 13:45:22.784861273 +0000 UTC m=+2490.930605101" Oct 31 13:45:22.803141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.803116 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc"] Oct 31 13:45:22.810257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.810213 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1ddade-29890905-252zc"] Oct 31 13:45:22.828292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.828269 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp"] Oct 31 13:45:22.838837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:22.838807 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-33c10a-29890905-9j8jp"] Oct 31 13:45:23.752742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:23.752700 5532 generic.go:356] "Generic (PLEG): container finished" podID="67cdbfbc-db13-43bd-823e-c1d61c8ec9e8" containerID="3445af2d9cff82b521d3fafaa79bcb56e3f67233c1b7f756e497858252fdd6a2" exitCode=0 Oct 31 13:45:23.753168 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:23.752783 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" event={"ID":"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8","Type":"ContainerDied","Data":"3445af2d9cff82b521d3fafaa79bcb56e3f67233c1b7f756e497858252fdd6a2"} Oct 31 13:45:23.945577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:23.945552 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" Oct 31 13:45:23.990098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:23.990061 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c65a8d-5c95-4356-954d-106740ce3181-kube-api-access-tc49r\" (UniqueName: \"kubernetes.io/projected/b9c65a8d-5c95-4356-954d-106740ce3181-kube-api-access-tc49r\") pod \"b9c65a8d-5c95-4356-954d-106740ce3181\" (UID: \"b9c65a8d-5c95-4356-954d-106740ce3181\") " Oct 31 13:45:23.990372 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:23.990355 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-openshift-ca\") pod \"b9c65a8d-5c95-4356-954d-106740ce3181\" (UID: \"b9c65a8d-5c95-4356-954d-106740ce3181\") " Oct 31 13:45:23.990467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:23.990393 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-xks-ca\") pod \"b9c65a8d-5c95-4356-954d-106740ce3181\" (UID: \"b9c65a8d-5c95-4356-954d-106740ce3181\") " Oct 31 13:45:23.990760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:23.990729 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-openshift-ca" pod "b9c65a8d-5c95-4356-954d-106740ce3181" (UID: "b9c65a8d-5c95-4356-954d-106740ce3181"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:23.991118 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:23.991097 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:23.992802 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:23.992776 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-xks-ca" pod "b9c65a8d-5c95-4356-954d-106740ce3181" (UID: "b9c65a8d-5c95-4356-954d-106740ce3181"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:23.992896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:23.992820 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c65a8d-5c95-4356-954d-106740ce3181-kube-api-access-tc49r" pod "b9c65a8d-5c95-4356-954d-106740ce3181" (UID: "b9c65a8d-5c95-4356-954d-106740ce3181"). InnerVolumeSpecName "kube-api-access-tc49r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:24.012909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:24.012879 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="096af709-7898-469d-a55b-bbbacf1cb43e" path="/var/lib/kubelet/pods/096af709-7898-469d-a55b-bbbacf1cb43e/volumes" Oct 31 13:45:24.013459 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:24.013442 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76565e81-762a-4d6d-b887-9c75556cbed3" path="/var/lib/kubelet/pods/76565e81-762a-4d6d-b887-9c75556cbed3/volumes" Oct 31 13:45:24.013997 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:24.013977 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" path="/var/lib/kubelet/pods/7f805bdf-0bd7-4ea9-8378-18f10dbc7d98/volumes" Oct 31 13:45:24.092403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:24.092375 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9c65a8d-5c95-4356-954d-106740ce3181-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:24.092554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:24.092468 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc49r\" (UniqueName: \"kubernetes.io/projected/b9c65a8d-5c95-4356-954d-106740ce3181-kube-api-access-tc49r\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:24.764776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:24.764741 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" Oct 31 13:45:24.765238 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:24.764774 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w" event={"ID":"b9c65a8d-5c95-4356-954d-106740ce3181","Type":"ContainerDied","Data":"2fcb2d810411361d04e2e4842999ddbf9350ea1e6db2f8e6d5a941668b1a76a6"} Oct 31 13:45:24.765238 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:24.764835 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2fcb2d810411361d04e2e4842999ddbf9350ea1e6db2f8e6d5a941668b1a76a6" Oct 31 13:45:24.960935 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:24.960911 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" Oct 31 13:45:25.009588 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:25.009559 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-openshift-ca\") pod \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\" (UID: \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\") " Oct 31 13:45:25.009811 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:25.009780 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-kube-api-access-jm97r\" (UniqueName: \"kubernetes.io/projected/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-kube-api-access-jm97r\") pod \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\" (UID: \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\") " Oct 31 13:45:25.009922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:25.009906 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-xks-ca\") pod \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\" (UID: \"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8\") " Oct 31 13:45:25.009995 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:25.009934 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-openshift-ca" pod "67cdbfbc-db13-43bd-823e-c1d61c8ec9e8" (UID: "67cdbfbc-db13-43bd-823e-c1d61c8ec9e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:25.010753 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:25.010727 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:25.012212 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:25.012189 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-kube-api-access-jm97r" pod "67cdbfbc-db13-43bd-823e-c1d61c8ec9e8" (UID: "67cdbfbc-db13-43bd-823e-c1d61c8ec9e8"). InnerVolumeSpecName "kube-api-access-jm97r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:25.012760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:25.012735 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-xks-ca" pod "67cdbfbc-db13-43bd-823e-c1d61c8ec9e8" (UID: "67cdbfbc-db13-43bd-823e-c1d61c8ec9e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:25.112472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:25.112433 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm97r\" (UniqueName: \"kubernetes.io/projected/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-kube-api-access-jm97r\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:25.112472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:25.112466 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:25.780463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:25.780417 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" Oct 31 13:45:25.780950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:25.780648 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf" event={"ID":"67cdbfbc-db13-43bd-823e-c1d61c8ec9e8","Type":"ContainerDied","Data":"317f105681d0b543edce0c0b057a192630924950eb295111124645a4001386cf"} Oct 31 13:45:25.780950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:25.780702 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="317f105681d0b543edce0c0b057a192630924950eb295111124645a4001386cf" Oct 31 13:45:30.977127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:30.977074 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt"] Oct 31 13:45:30.977728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:30.977373 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" podUID="5a9332ef-beec-4b46-9617-36d8ce8812c5" containerName="istio-proxy" containerID="cri-o://a443ce7ba02030173423306a865f35938863803713af01ee5e84116b6e294410" gracePeriod=30 Oct 31 13:45:31.194871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:31.194838 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd"] Oct 31 13:45:31.195052 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:31.195032 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" containerID="cri-o://8754d56bb37d4beace8f260f1de071b9bcba31a2de693b0fe4fce6c426dd261b" gracePeriod=30 Oct 31 13:45:31.353983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:31.353897 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j"] Oct 31 13:45:31.354178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:31.354155 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" containerID="cri-o://489352a06c6a12945cd6c093196654f693d8f171d20455c48047cc8ab7de1673" gracePeriod=30 Oct 31 13:45:31.468742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:31.468699 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5"] Oct 31 13:45:31.469784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:31.469046 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" containerID="cri-o://7a1a6c45a407e2a5cd51815da7caf99dc68d8aaf53fcba4622627e0ea5836467" gracePeriod=30 Oct 31 13:45:31.928616 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:31.927764 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:45:33.817906 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:33.817870 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz"] Oct 31 13:45:35.416270 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:35.416231 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf"] Oct 31 13:45:35.421382 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:35.421355 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb50fa-m7dtf"] Oct 31 13:45:36.009746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.009716 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67cdbfbc-db13-43bd-823e-c1d61c8ec9e8" path="/var/lib/kubelet/pods/67cdbfbc-db13-43bd-823e-c1d61c8ec9e8/volumes" Oct 31 13:45:36.331520 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.331494 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:45:36.425638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.424895 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-token\" (UniqueName: \"kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-token\") pod \"5a9332ef-beec-4b46-9617-36d8ce8812c5\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " Oct 31 13:45:36.425638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.425038 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-socket\") pod \"5a9332ef-beec-4b46-9617-36d8ce8812c5\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " Oct 31 13:45:36.425638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.425093 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-data\") pod \"5a9332ef-beec-4b46-9617-36d8ce8812c5\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " Oct 31 13:45:36.425638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.425154 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-envoy\") pod \"5a9332ef-beec-4b46-9617-36d8ce8812c5\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " Oct 31 13:45:36.425638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.425219 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-kube-api-access-skbk4\" (UniqueName: \"kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-kube-api-access-skbk4\") pod \"5a9332ef-beec-4b46-9617-36d8ce8812c5\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " Oct 31 13:45:36.425638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.425261 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-credential-socket\") pod \"5a9332ef-beec-4b46-9617-36d8ce8812c5\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " Oct 31 13:45:36.425638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.425306 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a9332ef-beec-4b46-9617-36d8ce8812c5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9332ef-beec-4b46-9617-36d8ce8812c5-istiod-ca-cert\") pod \"5a9332ef-beec-4b46-9617-36d8ce8812c5\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " Oct 31 13:45:36.425638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.425345 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-podinfo\") pod \"5a9332ef-beec-4b46-9617-36d8ce8812c5\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " Oct 31 13:45:36.426440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.425671 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-certs\") pod \"5a9332ef-beec-4b46-9617-36d8ce8812c5\" (UID: \"5a9332ef-beec-4b46-9617-36d8ce8812c5\") " Oct 31 13:45:36.427312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.426853 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-certs" pod "5a9332ef-beec-4b46-9617-36d8ce8812c5" (UID: "5a9332ef-beec-4b46-9617-36d8ce8812c5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.427312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.427046 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-socket" pod "5a9332ef-beec-4b46-9617-36d8ce8812c5" (UID: "5a9332ef-beec-4b46-9617-36d8ce8812c5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.427312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.427221 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-data" pod "5a9332ef-beec-4b46-9617-36d8ce8812c5" (UID: "5a9332ef-beec-4b46-9617-36d8ce8812c5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.428635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.427820 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-token" pod "5a9332ef-beec-4b46-9617-36d8ce8812c5" (UID: "5a9332ef-beec-4b46-9617-36d8ce8812c5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:36.428635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.428469 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a9332ef-beec-4b46-9617-36d8ce8812c5-istiod-ca-cert" pod "5a9332ef-beec-4b46-9617-36d8ce8812c5" (UID: "5a9332ef-beec-4b46-9617-36d8ce8812c5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:36.428819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.428717 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-credential-socket" pod "5a9332ef-beec-4b46-9617-36d8ce8812c5" (UID: "5a9332ef-beec-4b46-9617-36d8ce8812c5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.430650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.430232 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-envoy" pod "5a9332ef-beec-4b46-9617-36d8ce8812c5" (UID: "5a9332ef-beec-4b46-9617-36d8ce8812c5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.430842 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.430821 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-podinfo" pod "5a9332ef-beec-4b46-9617-36d8ce8812c5" (UID: "5a9332ef-beec-4b46-9617-36d8ce8812c5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:45:36.431522 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.431494 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-kube-api-access-skbk4" pod "5a9332ef-beec-4b46-9617-36d8ce8812c5" (UID: "5a9332ef-beec-4b46-9617-36d8ce8812c5"). InnerVolumeSpecName "kube-api-access-skbk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:36.528316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.528282 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.528316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.528317 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.528536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.528331 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.528536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.528342 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.528536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.528355 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skbk4\" (UniqueName: \"kubernetes.io/projected/5a9332ef-beec-4b46-9617-36d8ce8812c5-kube-api-access-skbk4\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.528536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.528368 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.528536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.528380 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a9332ef-beec-4b46-9617-36d8ce8812c5-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.528536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.528393 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a9332ef-beec-4b46-9617-36d8ce8812c5-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.528536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.528405 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a9332ef-beec-4b46-9617-36d8ce8812c5-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.640693 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.639216 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:45:36.736994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.731815 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-podinfo\") pod \"a3f61058-6f46-454c-a7a6-fea17390fc1c\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " Oct 31 13:45:36.736994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.731872 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-envoy\") pod \"a3f61058-6f46-454c-a7a6-fea17390fc1c\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " Oct 31 13:45:36.736994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.731998 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-kube-api-access-hbj2n\" (UniqueName: \"kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-kube-api-access-hbj2n\") pod \"a3f61058-6f46-454c-a7a6-fea17390fc1c\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " Oct 31 13:45:36.736994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.732046 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-credential-socket\") pod \"a3f61058-6f46-454c-a7a6-fea17390fc1c\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " Oct 31 13:45:36.736994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.732098 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-data\") pod \"a3f61058-6f46-454c-a7a6-fea17390fc1c\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " Oct 31 13:45:36.736994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.732205 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-certs\") pod \"a3f61058-6f46-454c-a7a6-fea17390fc1c\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " Oct 31 13:45:36.736994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.732367 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f61058-6f46-454c-a7a6-fea17390fc1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f61058-6f46-454c-a7a6-fea17390fc1c-istiod-ca-cert\") pod \"a3f61058-6f46-454c-a7a6-fea17390fc1c\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " Oct 31 13:45:36.736994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.732489 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-token\" (UniqueName: \"kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-token\") pod \"a3f61058-6f46-454c-a7a6-fea17390fc1c\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " Oct 31 13:45:36.736994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.732521 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-socket\") pod \"a3f61058-6f46-454c-a7a6-fea17390fc1c\" (UID: \"a3f61058-6f46-454c-a7a6-fea17390fc1c\") " Oct 31 13:45:36.736994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.733740 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-socket" pod "a3f61058-6f46-454c-a7a6-fea17390fc1c" (UID: "a3f61058-6f46-454c-a7a6-fea17390fc1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.737564 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.737245 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-certs" pod "a3f61058-6f46-454c-a7a6-fea17390fc1c" (UID: "a3f61058-6f46-454c-a7a6-fea17390fc1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.739930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.739897 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-kube-api-access-hbj2n" pod "a3f61058-6f46-454c-a7a6-fea17390fc1c" (UID: "a3f61058-6f46-454c-a7a6-fea17390fc1c"). InnerVolumeSpecName "kube-api-access-hbj2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:36.740080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.739971 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-envoy" pod "a3f61058-6f46-454c-a7a6-fea17390fc1c" (UID: "a3f61058-6f46-454c-a7a6-fea17390fc1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.740411 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.740357 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-credential-socket" pod "a3f61058-6f46-454c-a7a6-fea17390fc1c" (UID: "a3f61058-6f46-454c-a7a6-fea17390fc1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.741971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.741947 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f61058-6f46-454c-a7a6-fea17390fc1c-istiod-ca-cert" pod "a3f61058-6f46-454c-a7a6-fea17390fc1c" (UID: "a3f61058-6f46-454c-a7a6-fea17390fc1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:36.742256 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.742230 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-token" pod "a3f61058-6f46-454c-a7a6-fea17390fc1c" (UID: "a3f61058-6f46-454c-a7a6-fea17390fc1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:36.742631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.742610 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-data" pod "a3f61058-6f46-454c-a7a6-fea17390fc1c" (UID: "a3f61058-6f46-454c-a7a6-fea17390fc1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.743310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.743289 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-podinfo" pod "a3f61058-6f46-454c-a7a6-fea17390fc1c" (UID: "a3f61058-6f46-454c-a7a6-fea17390fc1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:45:36.806649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.806615 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:45:36.840749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.836397 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f61058-6f46-454c-a7a6-fea17390fc1c-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.840749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.836436 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.840749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.836455 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.840749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.836470 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.840749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.836487 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.840749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.836501 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbj2n\" (UniqueName: \"kubernetes.io/projected/a3f61058-6f46-454c-a7a6-fea17390fc1c-kube-api-access-hbj2n\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.840749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.836516 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.840749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.836533 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.840749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.836547 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f61058-6f46-454c-a7a6-fea17390fc1c-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.937917 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.937881 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-podinfo\") pod \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " Oct 31 13:45:36.938097 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.937999 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-envoy\") pod \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " Oct 31 13:45:36.938097 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.938081 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-socket\") pod \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " Oct 31 13:45:36.938221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.938109 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-kube-api-access-bwcnk\" (UniqueName: \"kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-kube-api-access-bwcnk\") pod \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " Oct 31 13:45:36.938221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.938183 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-credential-socket\") pod \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " Oct 31 13:45:36.938388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.938356 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-certs\") pod \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " Oct 31 13:45:36.938463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.938426 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-data\") pod \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " Oct 31 13:45:36.938521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.938489 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istiod-ca-cert\") pod \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " Oct 31 13:45:36.938572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.938523 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-token\" (UniqueName: \"kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-token\") pod \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\" (UID: \"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd\") " Oct 31 13:45:36.938924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.938899 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-credential-socket" pod "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" (UID: "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.955235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.939697 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-data" pod "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" (UID: "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.955235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.940091 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istiod-ca-cert" pod "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" (UID: "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:36.955235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.940161 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-certs" pod "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" (UID: "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.955235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.940213 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-socket" pod "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" (UID: "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.956355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.956225 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.956355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.956265 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.956355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.956282 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.956355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.956296 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.956355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.956310 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:36.979099 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.964178 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-kube-api-access-bwcnk" pod "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" (UID: "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd"). InnerVolumeSpecName "kube-api-access-bwcnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:36.979099 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.971858 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-token" pod "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" (UID: "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:36.996757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.996091 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-envoy" pod "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" (UID: "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:36.997155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.997124 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-podinfo" pod "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" (UID: "8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:45:36.998057 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.998031 5532 generic.go:356] "Generic (PLEG): container finished" podID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerID="7a1a6c45a407e2a5cd51815da7caf99dc68d8aaf53fcba4622627e0ea5836467" exitCode=0 Oct 31 13:45:36.998261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:36.998203 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" event={"ID":"8bf67cc4-ed1a-4169-957b-0fe690432386","Type":"ContainerDied","Data":"7a1a6c45a407e2a5cd51815da7caf99dc68d8aaf53fcba4622627e0ea5836467"} Oct 31 13:45:37.000368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.000342 5532 generic.go:356] "Generic (PLEG): container finished" podID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerID="8754d56bb37d4beace8f260f1de071b9bcba31a2de693b0fe4fce6c426dd261b" exitCode=0 Oct 31 13:45:37.000560 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.000546 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" Oct 31 13:45:37.001167 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.001141 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" event={"ID":"a3f61058-6f46-454c-a7a6-fea17390fc1c","Type":"ContainerDied","Data":"8754d56bb37d4beace8f260f1de071b9bcba31a2de693b0fe4fce6c426dd261b"} Oct 31 13:45:37.001279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.001192 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd" event={"ID":"a3f61058-6f46-454c-a7a6-fea17390fc1c","Type":"ContainerDied","Data":"9e2ae0b04836137bb8e5b3f9fbd40d89d6e5fb6498491641140de3a5131d695f"} Oct 31 13:45:37.001279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.001243 5532 scope.go:122] "RemoveContainer" containerID="8754d56bb37d4beace8f260f1de071b9bcba31a2de693b0fe4fce6c426dd261b" Oct 31 13:45:37.005266 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.005243 5532 generic.go:356] "Generic (PLEG): container finished" podID="5a9332ef-beec-4b46-9617-36d8ce8812c5" containerID="a443ce7ba02030173423306a865f35938863803713af01ee5e84116b6e294410" exitCode=0 Oct 31 13:45:37.005378 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.005355 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" event={"ID":"5a9332ef-beec-4b46-9617-36d8ce8812c5","Type":"ContainerDied","Data":"a443ce7ba02030173423306a865f35938863803713af01ee5e84116b6e294410"} Oct 31 13:45:37.005459 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.005392 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" event={"ID":"5a9332ef-beec-4b46-9617-36d8ce8812c5","Type":"ContainerDied","Data":"a2d1fb616db1b64ab6bc274cb4e89d7e27157509f9f56ef40c4013531d7066fb"} Oct 31 13:45:37.005524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.005458 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt" Oct 31 13:45:37.012311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.012283 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:45:37.015912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.015887 5532 generic.go:356] "Generic (PLEG): container finished" podID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerID="489352a06c6a12945cd6c093196654f693d8f171d20455c48047cc8ab7de1673" exitCode=0 Oct 31 13:45:37.016015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.015943 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" event={"ID":"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd","Type":"ContainerDied","Data":"489352a06c6a12945cd6c093196654f693d8f171d20455c48047cc8ab7de1673"} Oct 31 13:45:37.016015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.015977 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" Oct 31 13:45:37.016140 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.015981 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j" event={"ID":"8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd","Type":"ContainerDied","Data":"e84fe3308f16344e0dc8d0ac096753cc4cc5412a56dfd0059c1a02bf0fed36d2"} Oct 31 13:45:37.058123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.058025 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:37.058123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.058074 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:37.058123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.058092 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwcnk\" (UniqueName: \"kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-kube-api-access-bwcnk\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:37.058123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.058107 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:37.073692 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.073665 5532 scope.go:122] "RemoveContainer" containerID="8754d56bb37d4beace8f260f1de071b9bcba31a2de693b0fe4fce6c426dd261b" Oct 31 13:45:37.074224 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:37.074201 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8754d56bb37d4beace8f260f1de071b9bcba31a2de693b0fe4fce6c426dd261b\": container with ID starting with 8754d56bb37d4beace8f260f1de071b9bcba31a2de693b0fe4fce6c426dd261b not found: ID does not exist" containerID="8754d56bb37d4beace8f260f1de071b9bcba31a2de693b0fe4fce6c426dd261b" Oct 31 13:45:37.074331 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.074230 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8754d56bb37d4beace8f260f1de071b9bcba31a2de693b0fe4fce6c426dd261b"} err="failed to get container status \"8754d56bb37d4beace8f260f1de071b9bcba31a2de693b0fe4fce6c426dd261b\": rpc error: code = NotFound desc = could not find container \"8754d56bb37d4beace8f260f1de071b9bcba31a2de693b0fe4fce6c426dd261b\": container with ID starting with 8754d56bb37d4beace8f260f1de071b9bcba31a2de693b0fe4fce6c426dd261b not found: ID does not exist" Oct 31 13:45:37.074331 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.074250 5532 scope.go:122] "RemoveContainer" containerID="a443ce7ba02030173423306a865f35938863803713af01ee5e84116b6e294410" Oct 31 13:45:37.099870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.099849 5532 scope.go:122] "RemoveContainer" containerID="a443ce7ba02030173423306a865f35938863803713af01ee5e84116b6e294410" Oct 31 13:45:37.100366 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:37.100344 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a443ce7ba02030173423306a865f35938863803713af01ee5e84116b6e294410\": container with ID starting with a443ce7ba02030173423306a865f35938863803713af01ee5e84116b6e294410 not found: ID does not exist" containerID="a443ce7ba02030173423306a865f35938863803713af01ee5e84116b6e294410" Oct 31 13:45:37.100496 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.100474 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a443ce7ba02030173423306a865f35938863803713af01ee5e84116b6e294410"} err="failed to get container status \"a443ce7ba02030173423306a865f35938863803713af01ee5e84116b6e294410\": rpc error: code = NotFound desc = could not find container \"a443ce7ba02030173423306a865f35938863803713af01ee5e84116b6e294410\": container with ID starting with a443ce7ba02030173423306a865f35938863803713af01ee5e84116b6e294410 not found: ID does not exist" Oct 31 13:45:37.100575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.100565 5532 scope.go:122] "RemoveContainer" containerID="489352a06c6a12945cd6c093196654f693d8f171d20455c48047cc8ab7de1673" Oct 31 13:45:37.105206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.105187 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt"] Oct 31 13:45:37.113340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.113287 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-hq6tt"] Oct 31 13:45:37.122226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.122176 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd"] Oct 31 13:45:37.130493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.130469 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-mqwqd"] Oct 31 13:45:37.142657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.142266 5532 scope.go:122] "RemoveContainer" containerID="489352a06c6a12945cd6c093196654f693d8f171d20455c48047cc8ab7de1673" Oct 31 13:45:37.142791 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:37.142723 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489352a06c6a12945cd6c093196654f693d8f171d20455c48047cc8ab7de1673\": container with ID starting with 489352a06c6a12945cd6c093196654f693d8f171d20455c48047cc8ab7de1673 not found: ID does not exist" containerID="489352a06c6a12945cd6c093196654f693d8f171d20455c48047cc8ab7de1673" Oct 31 13:45:37.142791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.142759 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489352a06c6a12945cd6c093196654f693d8f171d20455c48047cc8ab7de1673"} err="failed to get container status \"489352a06c6a12945cd6c093196654f693d8f171d20455c48047cc8ab7de1673\": rpc error: code = NotFound desc = could not find container \"489352a06c6a12945cd6c093196654f693d8f171d20455c48047cc8ab7de1673\": container with ID starting with 489352a06c6a12945cd6c093196654f693d8f171d20455c48047cc8ab7de1673 not found: ID does not exist" Oct 31 13:45:37.145763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.145708 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j"] Oct 31 13:45:37.159157 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.159104 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-podinfo\") pod \"8bf67cc4-ed1a-4169-957b-0fe690432386\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " Oct 31 13:45:37.159543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.159480 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf67cc4-ed1a-4169-957b-0fe690432386-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf67cc4-ed1a-4169-957b-0fe690432386-istiod-ca-cert\") pod \"8bf67cc4-ed1a-4169-957b-0fe690432386\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " Oct 31 13:45:37.159928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.159832 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-socket\") pod \"8bf67cc4-ed1a-4169-957b-0fe690432386\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " Oct 31 13:45:37.160164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.160101 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-token\" (UniqueName: \"kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-token\") pod \"8bf67cc4-ed1a-4169-957b-0fe690432386\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " Oct 31 13:45:37.161138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.160387 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-credential-socket\") pod \"8bf67cc4-ed1a-4169-957b-0fe690432386\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " Oct 31 13:45:37.161138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.160456 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-kube-api-access-cgvqw\" (UniqueName: \"kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-kube-api-access-cgvqw\") pod \"8bf67cc4-ed1a-4169-957b-0fe690432386\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " Oct 31 13:45:37.161138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.160526 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-certs\") pod \"8bf67cc4-ed1a-4169-957b-0fe690432386\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " Oct 31 13:45:37.161138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.160734 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-data\") pod \"8bf67cc4-ed1a-4169-957b-0fe690432386\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " Oct 31 13:45:37.161138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.160777 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-envoy\") pod \"8bf67cc4-ed1a-4169-957b-0fe690432386\" (UID: \"8bf67cc4-ed1a-4169-957b-0fe690432386\") " Oct 31 13:45:37.164910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.164065 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-certs" pod "8bf67cc4-ed1a-4169-957b-0fe690432386" (UID: "8bf67cc4-ed1a-4169-957b-0fe690432386"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:37.164910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.164284 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-socket" pod "8bf67cc4-ed1a-4169-957b-0fe690432386" (UID: "8bf67cc4-ed1a-4169-957b-0fe690432386"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:37.164910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.164469 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-credential-socket" pod "8bf67cc4-ed1a-4169-957b-0fe690432386" (UID: "8bf67cc4-ed1a-4169-957b-0fe690432386"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:37.164910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.164757 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf67cc4-ed1a-4169-957b-0fe690432386-istiod-ca-cert" pod "8bf67cc4-ed1a-4169-957b-0fe690432386" (UID: "8bf67cc4-ed1a-4169-957b-0fe690432386"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:37.164910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.164876 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-data" pod "8bf67cc4-ed1a-4169-957b-0fe690432386" (UID: "8bf67cc4-ed1a-4169-957b-0fe690432386"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:37.165420 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.165400 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-9g96j"] Oct 31 13:45:37.166373 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.166330 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-podinfo" pod "8bf67cc4-ed1a-4169-957b-0fe690432386" (UID: "8bf67cc4-ed1a-4169-957b-0fe690432386"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:45:37.167088 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.166869 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-token" pod "8bf67cc4-ed1a-4169-957b-0fe690432386" (UID: "8bf67cc4-ed1a-4169-957b-0fe690432386"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:37.167563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.167524 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-envoy" pod "8bf67cc4-ed1a-4169-957b-0fe690432386" (UID: "8bf67cc4-ed1a-4169-957b-0fe690432386"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:37.168784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.168763 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-kube-api-access-cgvqw" pod "8bf67cc4-ed1a-4169-957b-0fe690432386" (UID: "8bf67cc4-ed1a-4169-957b-0fe690432386"). InnerVolumeSpecName "kube-api-access-cgvqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:37.264338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.264306 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:37.264338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.264334 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:37.264338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.264345 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgvqw\" (UniqueName: \"kubernetes.io/projected/8bf67cc4-ed1a-4169-957b-0fe690432386-kube-api-access-cgvqw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:37.264338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.264355 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:37.264622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.264364 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:37.264622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.264373 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:37.264622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.264382 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bf67cc4-ed1a-4169-957b-0fe690432386-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:37.264622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.264391 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bf67cc4-ed1a-4169-957b-0fe690432386-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:37.264622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:37.264400 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bf67cc4-ed1a-4169-957b-0fe690432386-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:38.007523 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:38.007490 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a9332ef-beec-4b46-9617-36d8ce8812c5" path="/var/lib/kubelet/pods/5a9332ef-beec-4b46-9617-36d8ce8812c5/volumes" Oct 31 13:45:38.008072 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:38.008050 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" path="/var/lib/kubelet/pods/8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd/volumes" Oct 31 13:45:38.008537 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:38.008523 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" path="/var/lib/kubelet/pods/a3f61058-6f46-454c-a7a6-fea17390fc1c/volumes" Oct 31 13:45:38.026903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:38.026883 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" Oct 31 13:45:38.027037 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:38.026916 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5" event={"ID":"8bf67cc4-ed1a-4169-957b-0fe690432386","Type":"ContainerDied","Data":"8e1cee708f2adcf3ed027ea8c47fe029ef59b15e05a1eadb78ecd9e3ee737b03"} Oct 31 13:45:38.027037 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:38.026971 5532 scope.go:122] "RemoveContainer" containerID="7a1a6c45a407e2a5cd51815da7caf99dc68d8aaf53fcba4622627e0ea5836467" Oct 31 13:45:38.057356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:38.057332 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5"] Oct 31 13:45:38.064525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:38.064491 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-7n9t5"] Oct 31 13:45:40.007409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:40.007373 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" path="/var/lib/kubelet/pods/8bf67cc4-ed1a-4169-957b-0fe690432386/volumes" Oct 31 13:45:40.117789 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:40.117753 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj"] Oct 31 13:45:48.490704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.490666 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm"] Oct 31 13:45:48.491529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491512 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c810f088-8712-4c24-bc79-1e19589746bb" containerName="cleanup" Oct 31 13:45:48.491574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491531 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c810f088-8712-4c24-bc79-1e19589746bb" containerName="cleanup" Oct 31 13:45:48.491574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491544 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" Oct 31 13:45:48.491574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491549 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" Oct 31 13:45:48.491574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491566 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b9c65a8d-5c95-4356-954d-106740ce3181" containerName="revoke-keys" Oct 31 13:45:48.491574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491571 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c65a8d-5c95-4356-954d-106740ce3181" containerName="revoke-keys" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491580 5532 cpu_manager.go:414] "containerMap: removing container" podUID="67cdbfbc-db13-43bd-823e-c1d61c8ec9e8" containerName="revoke-keys" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491585 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67cdbfbc-db13-43bd-823e-c1d61c8ec9e8" containerName="revoke-keys" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491613 5532 cpu_manager.go:414] "containerMap: removing container" podUID="096af709-7898-469d-a55b-bbbacf1cb43e" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491618 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096af709-7898-469d-a55b-bbbacf1cb43e" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491630 5532 cpu_manager.go:414] "containerMap: removing container" podUID="096af709-7898-469d-a55b-bbbacf1cb43e" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491634 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096af709-7898-469d-a55b-bbbacf1cb43e" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491644 5532 cpu_manager.go:414] "containerMap: removing container" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491648 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491658 5532 cpu_manager.go:414] "containerMap: removing container" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491663 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491674 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491680 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491687 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491691 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491702 5532 cpu_manager.go:414] "containerMap: removing container" podUID="096af709-7898-469d-a55b-bbbacf1cb43e" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491706 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="096af709-7898-469d-a55b-bbbacf1cb43e" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491714 5532 cpu_manager.go:414] "containerMap: removing container" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491719 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491726 5532 cpu_manager.go:414] "containerMap: removing container" podUID="76565e81-762a-4d6d-b887-9c75556cbed3" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491730 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76565e81-762a-4d6d-b887-9c75556cbed3" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491737 5532 cpu_manager.go:414] "containerMap: removing container" podUID="76565e81-762a-4d6d-b887-9c75556cbed3" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491741 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76565e81-762a-4d6d-b887-9c75556cbed3" containerName="cleanup" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491754 5532 cpu_manager.go:414] "containerMap: removing container" podUID="7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" containerName="istio-proxy" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491758 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" containerName="istio-proxy" Oct 31 13:45:48.491758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491766 5532 cpu_manager.go:414] "containerMap: removing container" podUID="5a9332ef-beec-4b46-9617-36d8ce8812c5" containerName="istio-proxy" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491771 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a9332ef-beec-4b46-9617-36d8ce8812c5" containerName="istio-proxy" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491779 5532 cpu_manager.go:414] "containerMap: removing container" podUID="76565e81-762a-4d6d-b887-9c75556cbed3" containerName="cleanup" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491784 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76565e81-762a-4d6d-b887-9c75556cbed3" containerName="cleanup" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491897 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f61058-6f46-454c-a7a6-fea17390fc1c" containerName="istio-proxy" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491906 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c65a8d-5c95-4356-954d-106740ce3181" containerName="revoke-keys" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491912 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="67cdbfbc-db13-43bd-823e-c1d61c8ec9e8" containerName="revoke-keys" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491921 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="096af709-7898-469d-a55b-bbbacf1cb43e" containerName="cleanup" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491928 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ead7008d-2414-412b-b024-2c4ddeda9399" containerName="cleanup" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491935 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d1da6f9-c84f-4b0d-a805-7f3baa5b0cbd" containerName="istio-proxy" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491941 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd2e824-e299-438a-ae40-d3ebc513943d" containerName="cleanup" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491947 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d93b407-6ae0-44b8-adab-96c1ed5b6d2b" containerName="cleanup" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491953 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="096af709-7898-469d-a55b-bbbacf1cb43e" containerName="cleanup" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491958 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf67cc4-ed1a-4169-957b-0fe690432386" containerName="istio-proxy" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491963 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="76565e81-762a-4d6d-b887-9c75556cbed3" containerName="cleanup" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491969 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="76565e81-762a-4d6d-b887-9c75556cbed3" containerName="cleanup" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491975 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f805bdf-0bd7-4ea9-8378-18f10dbc7d98" containerName="istio-proxy" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491981 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a9332ef-beec-4b46-9617-36d8ce8812c5" containerName="istio-proxy" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491987 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c810f088-8712-4c24-bc79-1e19589746bb" containerName="cleanup" Oct 31 13:45:48.492518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.491993 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="76565e81-762a-4d6d-b887-9c75556cbed3" containerName="cleanup" Oct 31 13:45:48.496271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.496250 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" Oct 31 13:45:48.501850 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.501819 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:45:48.502578 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.502544 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:45:48.507853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.507829 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm"] Oct 31 13:45:48.525977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.525939 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm\" (UID: \"4a213358-40d2-4ded-9697-d6b2739fc99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" Oct 31 13:45:48.526288 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.526256 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556h2\" (UniqueName: \"kubernetes.io/projected/4a213358-40d2-4ded-9697-d6b2739fc99b-kube-api-access-556h2\") pod \"maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm\" (UID: \"4a213358-40d2-4ded-9697-d6b2739fc99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" Oct 31 13:45:48.526440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.526419 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm\" (UID: \"4a213358-40d2-4ded-9697-d6b2739fc99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" Oct 31 13:45:48.627940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.627899 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556h2\" (UniqueName: \"kubernetes.io/projected/4a213358-40d2-4ded-9697-d6b2739fc99b-kube-api-access-556h2\") pod \"maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm\" (UID: \"4a213358-40d2-4ded-9697-d6b2739fc99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" Oct 31 13:45:48.628126 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.627993 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm\" (UID: \"4a213358-40d2-4ded-9697-d6b2739fc99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" Oct 31 13:45:48.628126 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.628086 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm\" (UID: \"4a213358-40d2-4ded-9697-d6b2739fc99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" Oct 31 13:45:48.628722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.628692 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm\" (UID: \"4a213358-40d2-4ded-9697-d6b2739fc99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" Oct 31 13:45:48.630429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.630409 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm\" (UID: \"4a213358-40d2-4ded-9697-d6b2739fc99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" Oct 31 13:45:48.672915 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.669890 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-556h2\" (UniqueName: \"kubernetes.io/projected/4a213358-40d2-4ded-9697-d6b2739fc99b-kube-api-access-556h2\") pod \"maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm\" (UID: \"4a213358-40d2-4ded-9697-d6b2739fc99b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" Oct 31 13:45:48.814705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.814615 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" Oct 31 13:45:48.840896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:48.840863 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" podUID="fb8e0c0b-e52a-4752-93e9-2624bbde361b" containerName="main" containerID="cri-o://a4b10044b91d165dc532806c2759a338ecda5686db68afc939d4ebf767370b36" gracePeriod=45 Oct 31 13:45:49.115391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.115365 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:45:49.120689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.120188 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm"] Oct 31 13:45:49.120689 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:45:49.120525 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a213358_40d2_4ded_9697_d6b2739fc99b.slice/crio-140b2089bf28adb6616f23dbaf5d2592669476601f2b5ab8ea6efb763cbf7d29 WatchSource:0}: Error finding container 140b2089bf28adb6616f23dbaf5d2592669476601f2b5ab8ea6efb763cbf7d29: Status 404 returned error can't find the container with id 140b2089bf28adb6616f23dbaf5d2592669476601f2b5ab8ea6efb763cbf7d29 Oct 31 13:45:49.136460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.136273 5532 generic.go:356] "Generic (PLEG): container finished" podID="fb8e0c0b-e52a-4752-93e9-2624bbde361b" containerID="a4b10044b91d165dc532806c2759a338ecda5686db68afc939d4ebf767370b36" exitCode=0 Oct 31 13:45:49.136460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.136418 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" event={"ID":"fb8e0c0b-e52a-4752-93e9-2624bbde361b","Type":"ContainerDied","Data":"a4b10044b91d165dc532806c2759a338ecda5686db68afc939d4ebf767370b36"} Oct 31 13:45:49.136460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.136458 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" event={"ID":"fb8e0c0b-e52a-4752-93e9-2624bbde361b","Type":"ContainerDied","Data":"975530d870a19e7f5549ebc1822ad07233fd894003e8d8aedf7a04f71769f2ff"} Oct 31 13:45:49.136460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.136481 5532 scope.go:122] "RemoveContainer" containerID="a4b10044b91d165dc532806c2759a338ecda5686db68afc939d4ebf767370b36" Oct 31 13:45:49.138294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.136658 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz" Oct 31 13:45:49.139193 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.139167 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tmp-dir\") pod \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " Oct 31 13:45:49.139346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.139288 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-home\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-home\") pod \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " Oct 31 13:45:49.140438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.139534 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-dshm\") pod \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " Oct 31 13:45:49.140438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.139571 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-model-cache\") pod \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " Oct 31 13:45:49.140438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.139715 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb8e0c0b-e52a-4752-93e9-2624bbde361b-kube-api-access-lf278\" (UniqueName: \"kubernetes.io/projected/fb8e0c0b-e52a-4752-93e9-2624bbde361b-kube-api-access-lf278\") pod \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " Oct 31 13:45:49.140438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.139805 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tls-certs\" (UniqueName: \"kubernetes.io/secret/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tls-certs\") pod \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\" (UID: \"fb8e0c0b-e52a-4752-93e9-2624bbde361b\") " Oct 31 13:45:49.141845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.141562 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-home" pod "fb8e0c0b-e52a-4752-93e9-2624bbde361b" (UID: "fb8e0c0b-e52a-4752-93e9-2624bbde361b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:49.141845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.141573 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-model-cache" pod "fb8e0c0b-e52a-4752-93e9-2624bbde361b" (UID: "fb8e0c0b-e52a-4752-93e9-2624bbde361b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:49.142083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.141931 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tmp-dir" pod "fb8e0c0b-e52a-4752-93e9-2624bbde361b" (UID: "fb8e0c0b-e52a-4752-93e9-2624bbde361b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:49.148936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.148907 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" event={"ID":"4a213358-40d2-4ded-9697-d6b2739fc99b","Type":"ContainerStarted","Data":"140b2089bf28adb6616f23dbaf5d2592669476601f2b5ab8ea6efb763cbf7d29"} Oct 31 13:45:49.150064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.150033 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb8e0c0b-e52a-4752-93e9-2624bbde361b-kube-api-access-lf278" pod "fb8e0c0b-e52a-4752-93e9-2624bbde361b" (UID: "fb8e0c0b-e52a-4752-93e9-2624bbde361b"). InnerVolumeSpecName "kube-api-access-lf278". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:49.150861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.150828 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-dshm" pod "fb8e0c0b-e52a-4752-93e9-2624bbde361b" (UID: "fb8e0c0b-e52a-4752-93e9-2624bbde361b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:49.151358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.151336 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tls-certs" pod "fb8e0c0b-e52a-4752-93e9-2624bbde361b" (UID: "fb8e0c0b-e52a-4752-93e9-2624bbde361b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:49.161557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.161531 5532 scope.go:122] "RemoveContainer" containerID="a4b10044b91d165dc532806c2759a338ecda5686db68afc939d4ebf767370b36" Oct 31 13:45:49.161919 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:49.161897 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4b10044b91d165dc532806c2759a338ecda5686db68afc939d4ebf767370b36\": container with ID starting with a4b10044b91d165dc532806c2759a338ecda5686db68afc939d4ebf767370b36 not found: ID does not exist" containerID="a4b10044b91d165dc532806c2759a338ecda5686db68afc939d4ebf767370b36" Oct 31 13:45:49.162007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.161929 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4b10044b91d165dc532806c2759a338ecda5686db68afc939d4ebf767370b36"} err="failed to get container status \"a4b10044b91d165dc532806c2759a338ecda5686db68afc939d4ebf767370b36\": rpc error: code = NotFound desc = could not find container \"a4b10044b91d165dc532806c2759a338ecda5686db68afc939d4ebf767370b36\": container with ID starting with a4b10044b91d165dc532806c2759a338ecda5686db68afc939d4ebf767370b36 not found: ID does not exist" Oct 31 13:45:49.242215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.242185 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:49.242215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.242213 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:49.242430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.242223 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf278\" (UniqueName: \"kubernetes.io/projected/fb8e0c0b-e52a-4752-93e9-2624bbde361b-kube-api-access-lf278\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:49.242430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.242235 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:49.242430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.242244 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:49.242430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.242252 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fb8e0c0b-e52a-4752-93e9-2624bbde361b-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:49.488934 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.488900 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz"] Oct 31 13:45:49.501038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:49.501008 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-62778a/test-model-62778a-kserve-78797649bf-957bz"] Oct 31 13:45:50.007812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:50.007729 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb8e0c0b-e52a-4752-93e9-2624bbde361b" path="/var/lib/kubelet/pods/fb8e0c0b-e52a-4752-93e9-2624bbde361b/volumes" Oct 31 13:45:50.163611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:50.163561 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" event={"ID":"4a213358-40d2-4ded-9697-d6b2739fc99b","Type":"ContainerStarted","Data":"b1173a54f1cd3a04c4c7476393a38424d98bfde31edc4e4cba0e9bd4e114f256"} Oct 31 13:45:50.189639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:50.189576 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" podStartSLOduration=2.189562571 podStartE2EDuration="2.189562571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:50.187233927 +0000 UTC m=+2518.332977755" watchObservedRunningTime="2026-10-31 13:45:50.189562571 +0000 UTC m=+2518.335306420" Oct 31 13:45:51.173703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:51.173646 5532 generic.go:356] "Generic (PLEG): container finished" podID="4a213358-40d2-4ded-9697-d6b2739fc99b" containerID="b1173a54f1cd3a04c4c7476393a38424d98bfde31edc4e4cba0e9bd4e114f256" exitCode=0 Oct 31 13:45:51.174124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:51.173740 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" event={"ID":"4a213358-40d2-4ded-9697-d6b2739fc99b","Type":"ContainerDied","Data":"b1173a54f1cd3a04c4c7476393a38424d98bfde31edc4e4cba0e9bd4e114f256"} Oct 31 13:45:51.435512 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:51.435430 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz"] Oct 31 13:45:52.361323 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:52.361295 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" Oct 31 13:45:52.416194 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:52.416165 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-openshift-ca\") pod \"4a213358-40d2-4ded-9697-d6b2739fc99b\" (UID: \"4a213358-40d2-4ded-9697-d6b2739fc99b\") " Oct 31 13:45:52.416385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:52.416323 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a213358-40d2-4ded-9697-d6b2739fc99b-kube-api-access-556h2\" (UniqueName: \"kubernetes.io/projected/4a213358-40d2-4ded-9697-d6b2739fc99b-kube-api-access-556h2\") pod \"4a213358-40d2-4ded-9697-d6b2739fc99b\" (UID: \"4a213358-40d2-4ded-9697-d6b2739fc99b\") " Oct 31 13:45:52.416458 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:52.416429 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-xks-ca\") pod \"4a213358-40d2-4ded-9697-d6b2739fc99b\" (UID: \"4a213358-40d2-4ded-9697-d6b2739fc99b\") " Oct 31 13:45:52.416578 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:52.416558 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-openshift-ca" pod "4a213358-40d2-4ded-9697-d6b2739fc99b" (UID: "4a213358-40d2-4ded-9697-d6b2739fc99b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:52.417401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:52.417369 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:52.419141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:52.419109 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-xks-ca" pod "4a213358-40d2-4ded-9697-d6b2739fc99b" (UID: "4a213358-40d2-4ded-9697-d6b2739fc99b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:52.419403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:52.419381 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a213358-40d2-4ded-9697-d6b2739fc99b-kube-api-access-556h2" pod "4a213358-40d2-4ded-9697-d6b2739fc99b" (UID: "4a213358-40d2-4ded-9697-d6b2739fc99b"). InnerVolumeSpecName "kube-api-access-556h2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:52.519450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:52.519366 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-556h2\" (UniqueName: \"kubernetes.io/projected/4a213358-40d2-4ded-9697-d6b2739fc99b-kube-api-access-556h2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:52.519450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:52.519400 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a213358-40d2-4ded-9697-d6b2739fc99b-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:53.192694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:53.192666 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" Oct 31 13:45:53.192694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:53.192668 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm" event={"ID":"4a213358-40d2-4ded-9697-d6b2739fc99b","Type":"ContainerDied","Data":"140b2089bf28adb6616f23dbaf5d2592669476601f2b5ab8ea6efb763cbf7d29"} Oct 31 13:45:53.192923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:53.192720 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="140b2089bf28adb6616f23dbaf5d2592669476601f2b5ab8ea6efb763cbf7d29" Oct 31 13:45:54.875491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.875382 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h"] Oct 31 13:45:54.876687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.876665 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4a213358-40d2-4ded-9697-d6b2739fc99b" containerName="revoke-keys" Oct 31 13:45:54.876687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.876685 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a213358-40d2-4ded-9697-d6b2739fc99b" containerName="revoke-keys" Oct 31 13:45:54.876875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.876753 5532 cpu_manager.go:414] "containerMap: removing container" podUID="fb8e0c0b-e52a-4752-93e9-2624bbde361b" containerName="main" Oct 31 13:45:54.876875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.876759 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb8e0c0b-e52a-4752-93e9-2624bbde361b" containerName="main" Oct 31 13:45:54.877003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.876890 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb8e0c0b-e52a-4752-93e9-2624bbde361b" containerName="main" Oct 31 13:45:54.877003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.876905 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a213358-40d2-4ded-9697-d6b2739fc99b" containerName="revoke-keys" Oct 31 13:45:54.877003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.876915 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="096af709-7898-469d-a55b-bbbacf1cb43e" containerName="cleanup" Oct 31 13:45:54.883042 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.883018 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h"] Oct 31 13:45:54.883184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.883142 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" Oct 31 13:45:54.889200 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.889174 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:45:54.889653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.889634 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:45:54.987902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.987868 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h\" (UID: \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" Oct 31 13:45:54.988076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.987996 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h\" (UID: \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" Oct 31 13:45:54.988258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:54.988239 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hntdn\" (UniqueName: \"kubernetes.io/projected/1629727f-2f6f-4d20-a9ae-52ad396a7d65-kube-api-access-hntdn\") pod \"maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h\" (UID: \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" Oct 31 13:45:55.090202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.090153 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hntdn\" (UniqueName: \"kubernetes.io/projected/1629727f-2f6f-4d20-a9ae-52ad396a7d65-kube-api-access-hntdn\") pod \"maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h\" (UID: \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" Oct 31 13:45:55.090431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.090298 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h\" (UID: \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" Oct 31 13:45:55.090431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.090379 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h\" (UID: \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" Oct 31 13:45:55.091090 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.091065 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h\" (UID: \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" Oct 31 13:45:55.092757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.092733 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h\" (UID: \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" Oct 31 13:45:55.106905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.106858 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hntdn\" (UniqueName: \"kubernetes.io/projected/1629727f-2f6f-4d20-a9ae-52ad396a7d65-kube-api-access-hntdn\") pod \"maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h\" (UID: \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" Oct 31 13:45:55.139717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.139651 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" podUID="d9d69e89-8948-4a02-a8c5-35f1466b7a3f" containerName="main" containerID="cri-o://5b6ba57eda2f2d2dd43387559fbdc832773a6f70269c9c71e094e86fa7797f22" gracePeriod=45 Oct 31 13:45:55.204603 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.204562 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" Oct 31 13:45:55.441581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.441547 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q"] Oct 31 13:45:55.458382 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.458352 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs"] Oct 31 13:45:55.494925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.494897 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:45:55.515286 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.515248 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h"] Oct 31 13:45:55.515692 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:45:55.515666 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1629727f_2f6f_4d20_a9ae_52ad396a7d65.slice/crio-9ff65e76c5390b93d1af85cfe4354eb75250f7463cf2b39e6b39bdfe76c6d1e6 WatchSource:0}: Error finding container 9ff65e76c5390b93d1af85cfe4354eb75250f7463cf2b39e6b39bdfe76c6d1e6: Status 404 returned error can't find the container with id 9ff65e76c5390b93d1af85cfe4354eb75250f7463cf2b39e6b39bdfe76c6d1e6 Oct 31 13:45:55.599894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.599858 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-dshm\") pod \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " Oct 31 13:45:55.600053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.599962 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-home\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-home\") pod \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " Oct 31 13:45:55.600053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.600035 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-kube-api-access-vcrrs\" (UniqueName: \"kubernetes.io/projected/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-kube-api-access-vcrrs\") pod \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " Oct 31 13:45:55.600187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.600091 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tls-certs\") pod \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " Oct 31 13:45:55.600285 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.600250 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-home" pod "d9d69e89-8948-4a02-a8c5-35f1466b7a3f" (UID: "d9d69e89-8948-4a02-a8c5-35f1466b7a3f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:55.600285 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.600274 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-model-cache\") pod \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " Oct 31 13:45:55.600529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.600495 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tmp-dir\") pod \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\" (UID: \"d9d69e89-8948-4a02-a8c5-35f1466b7a3f\") " Oct 31 13:45:55.600708 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.600666 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-model-cache" pod "d9d69e89-8948-4a02-a8c5-35f1466b7a3f" (UID: "d9d69e89-8948-4a02-a8c5-35f1466b7a3f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:55.600863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.600831 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tmp-dir" pod "d9d69e89-8948-4a02-a8c5-35f1466b7a3f" (UID: "d9d69e89-8948-4a02-a8c5-35f1466b7a3f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:55.601828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.601802 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:55.601947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.601831 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:55.601947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.601845 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:55.603338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.603309 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-dshm" pod "d9d69e89-8948-4a02-a8c5-35f1466b7a3f" (UID: "d9d69e89-8948-4a02-a8c5-35f1466b7a3f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:45:55.603426 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.603376 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-kube-api-access-vcrrs" pod "d9d69e89-8948-4a02-a8c5-35f1466b7a3f" (UID: "d9d69e89-8948-4a02-a8c5-35f1466b7a3f"). InnerVolumeSpecName "kube-api-access-vcrrs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:55.604046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.604025 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tls-certs" pod "d9d69e89-8948-4a02-a8c5-35f1466b7a3f" (UID: "d9d69e89-8948-4a02-a8c5-35f1466b7a3f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:55.703384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.703300 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:55.703384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.703327 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:55.703384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:55.703339 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vcrrs\" (UniqueName: \"kubernetes.io/projected/d9d69e89-8948-4a02-a8c5-35f1466b7a3f-kube-api-access-vcrrs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:56.220909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:56.220873 5532 generic.go:356] "Generic (PLEG): container finished" podID="d9d69e89-8948-4a02-a8c5-35f1466b7a3f" containerID="5b6ba57eda2f2d2dd43387559fbdc832773a6f70269c9c71e094e86fa7797f22" exitCode=0 Oct 31 13:45:56.221330 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:56.220921 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" Oct 31 13:45:56.221330 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:56.220951 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" event={"ID":"d9d69e89-8948-4a02-a8c5-35f1466b7a3f","Type":"ContainerDied","Data":"5b6ba57eda2f2d2dd43387559fbdc832773a6f70269c9c71e094e86fa7797f22"} Oct 31 13:45:56.221330 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:56.220993 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj" event={"ID":"d9d69e89-8948-4a02-a8c5-35f1466b7a3f","Type":"ContainerDied","Data":"484bc6fdf7f1b9f81e3a3bb0fcd7d1ed391673433a2b88d5820f89c33b854920"} Oct 31 13:45:56.221330 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:56.221011 5532 scope.go:122] "RemoveContainer" containerID="5b6ba57eda2f2d2dd43387559fbdc832773a6f70269c9c71e094e86fa7797f22" Oct 31 13:45:56.223407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:56.223378 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" event={"ID":"1629727f-2f6f-4d20-a9ae-52ad396a7d65","Type":"ContainerStarted","Data":"afe623623c3943d0770451baab3609b8a6026c2af012cfb2919f43b9e9f82247"} Oct 31 13:45:56.223558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:56.223432 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" event={"ID":"1629727f-2f6f-4d20-a9ae-52ad396a7d65","Type":"ContainerStarted","Data":"9ff65e76c5390b93d1af85cfe4354eb75250f7463cf2b39e6b39bdfe76c6d1e6"} Oct 31 13:45:56.231810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:56.231793 5532 scope.go:122] "RemoveContainer" containerID="5b6ba57eda2f2d2dd43387559fbdc832773a6f70269c9c71e094e86fa7797f22" Oct 31 13:45:56.232085 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:45:56.232063 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b6ba57eda2f2d2dd43387559fbdc832773a6f70269c9c71e094e86fa7797f22\": container with ID starting with 5b6ba57eda2f2d2dd43387559fbdc832773a6f70269c9c71e094e86fa7797f22 not found: ID does not exist" containerID="5b6ba57eda2f2d2dd43387559fbdc832773a6f70269c9c71e094e86fa7797f22" Oct 31 13:45:56.232156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:56.232095 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b6ba57eda2f2d2dd43387559fbdc832773a6f70269c9c71e094e86fa7797f22"} err="failed to get container status \"5b6ba57eda2f2d2dd43387559fbdc832773a6f70269c9c71e094e86fa7797f22\": rpc error: code = NotFound desc = could not find container \"5b6ba57eda2f2d2dd43387559fbdc832773a6f70269c9c71e094e86fa7797f22\": container with ID starting with 5b6ba57eda2f2d2dd43387559fbdc832773a6f70269c9c71e094e86fa7797f22 not found: ID does not exist" Oct 31 13:45:56.246435 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:56.246411 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj"] Oct 31 13:45:56.253717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:56.253693 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-61da9d/test-model-61da9d-kserve-5fbddc95f7-pxsnj"] Oct 31 13:45:56.272481 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:56.272405 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" podStartSLOduration=2.272393203 podStartE2EDuration="2.272393203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:45:56.2687635 +0000 UTC m=+2524.414507329" watchObservedRunningTime="2026-10-31 13:45:56.272393203 +0000 UTC m=+2524.418137075" Oct 31 13:45:57.234441 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:57.234408 5532 generic.go:356] "Generic (PLEG): container finished" podID="1629727f-2f6f-4d20-a9ae-52ad396a7d65" containerID="afe623623c3943d0770451baab3609b8a6026c2af012cfb2919f43b9e9f82247" exitCode=0 Oct 31 13:45:57.234957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:57.234499 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" event={"ID":"1629727f-2f6f-4d20-a9ae-52ad396a7d65","Type":"ContainerDied","Data":"afe623623c3943d0770451baab3609b8a6026c2af012cfb2919f43b9e9f82247"} Oct 31 13:45:57.855253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:57.855218 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:45:58.048617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:58.047142 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9d69e89-8948-4a02-a8c5-35f1466b7a3f" path="/var/lib/kubelet/pods/d9d69e89-8948-4a02-a8c5-35f1466b7a3f/volumes" Oct 31 13:45:58.441950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:58.441924 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" Oct 31 13:45:58.461733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:58.461707 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1629727f-2f6f-4d20-a9ae-52ad396a7d65-kube-api-access-hntdn\" (UniqueName: \"kubernetes.io/projected/1629727f-2f6f-4d20-a9ae-52ad396a7d65-kube-api-access-hntdn\") pod \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\" (UID: \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\") " Oct 31 13:45:58.461844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:58.461819 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-xks-ca\") pod \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\" (UID: \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\") " Oct 31 13:45:58.461983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:58.461969 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-openshift-ca\") pod \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\" (UID: \"1629727f-2f6f-4d20-a9ae-52ad396a7d65\") " Oct 31 13:45:58.462347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:58.462320 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-openshift-ca" pod "1629727f-2f6f-4d20-a9ae-52ad396a7d65" (UID: "1629727f-2f6f-4d20-a9ae-52ad396a7d65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:45:58.462758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:58.462741 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:58.463856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:58.463832 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1629727f-2f6f-4d20-a9ae-52ad396a7d65-kube-api-access-hntdn" pod "1629727f-2f6f-4d20-a9ae-52ad396a7d65" (UID: "1629727f-2f6f-4d20-a9ae-52ad396a7d65"). InnerVolumeSpecName "kube-api-access-hntdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:45:58.463856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:58.463850 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-xks-ca" pod "1629727f-2f6f-4d20-a9ae-52ad396a7d65" (UID: "1629727f-2f6f-4d20-a9ae-52ad396a7d65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:45:58.564579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:58.564500 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1629727f-2f6f-4d20-a9ae-52ad396a7d65-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:58.564579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:58.564531 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hntdn\" (UniqueName: \"kubernetes.io/projected/1629727f-2f6f-4d20-a9ae-52ad396a7d65-kube-api-access-hntdn\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:45:59.253119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:59.253091 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" Oct 31 13:45:59.253308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:59.253128 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h" event={"ID":"1629727f-2f6f-4d20-a9ae-52ad396a7d65","Type":"ContainerDied","Data":"9ff65e76c5390b93d1af85cfe4354eb75250f7463cf2b39e6b39bdfe76c6d1e6"} Oct 31 13:45:59.253308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:45:59.253191 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ff65e76c5390b93d1af85cfe4354eb75250f7463cf2b39e6b39bdfe76c6d1e6" Oct 31 13:46:00.453513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:00.453407 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" podUID="402e298e-b8a6-4cef-8fa4-a5e667a33ca4" containerName="payload-processing" containerID="cri-o://d82c143b76cf5752c264f05efa1a66bb7450be8cb88aa7be35e66401042f8f33" gracePeriod=25 Oct 31 13:46:00.458710 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:00.458658 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" podUID="df4f4e60-7bc8-4d83-a6eb-97812b694793" containerName="payload-pre-processing" containerID="cri-o://3236bb00f7e74a4b265cfacbd0f9ea3b2a71c52e143145e0843ab00f8bbc0619" gracePeriod=25 Oct 31 13:46:01.383949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:01.383901 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:46:01.395881 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:01.395840 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:46:01.418877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:01.418377 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:46:01.419066 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:01.418994 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:46:01.488498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:01.488025 5532 generic.go:356] "Generic (PLEG): container finished" podID="402e298e-b8a6-4cef-8fa4-a5e667a33ca4" containerID="d82c143b76cf5752c264f05efa1a66bb7450be8cb88aa7be35e66401042f8f33" exitCode=0 Oct 31 13:46:01.488498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:01.488359 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" event={"ID":"402e298e-b8a6-4cef-8fa4-a5e667a33ca4","Type":"ContainerDied","Data":"d82c143b76cf5752c264f05efa1a66bb7450be8cb88aa7be35e66401042f8f33"} Oct 31 13:46:01.606663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:01.606191 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" Oct 31 13:46:01.694574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:01.694543 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-kube-api-access-mgtj7\" (UniqueName: \"kubernetes.io/projected/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-kube-api-access-mgtj7\") pod \"402e298e-b8a6-4cef-8fa4-a5e667a33ca4\" (UID: \"402e298e-b8a6-4cef-8fa4-a5e667a33ca4\") " Oct 31 13:46:01.694764 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:01.694638 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-plugins-config-volume\") pod \"402e298e-b8a6-4cef-8fa4-a5e667a33ca4\" (UID: \"402e298e-b8a6-4cef-8fa4-a5e667a33ca4\") " Oct 31 13:46:01.695120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:01.695081 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-plugins-config-volume" pod "402e298e-b8a6-4cef-8fa4-a5e667a33ca4" (UID: "402e298e-b8a6-4cef-8fa4-a5e667a33ca4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:01.695637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:01.695572 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:01.697082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:01.697058 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-kube-api-access-mgtj7" pod "402e298e-b8a6-4cef-8fa4-a5e667a33ca4" (UID: "402e298e-b8a6-4cef-8fa4-a5e667a33ca4"). InnerVolumeSpecName "kube-api-access-mgtj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:01.783141 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:46:01.783042 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:46:01.797448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:01.797414 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgtj7\" (UniqueName: \"kubernetes.io/projected/402e298e-b8a6-4cef-8fa4-a5e667a33ca4-kube-api-access-mgtj7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:02.499134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:02.499096 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" event={"ID":"402e298e-b8a6-4cef-8fa4-a5e667a33ca4","Type":"ContainerDied","Data":"4876cd82a47517dacf1b6011c2dcca2e2796b4e63d61838cdfbd32dc8483c53b"} Oct 31 13:46:02.499527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:02.499157 5532 scope.go:122] "RemoveContainer" containerID="d82c143b76cf5752c264f05efa1a66bb7450be8cb88aa7be35e66401042f8f33" Oct 31 13:46:02.499527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:02.499267 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q" Oct 31 13:46:02.525371 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:02.525344 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q"] Oct 31 13:46:02.531084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:02.531057 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-61f295-58c5d7685c-2mf2q"] Oct 31 13:46:04.031981 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:04.022428 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="402e298e-b8a6-4cef-8fa4-a5e667a33ca4" path="/var/lib/kubelet/pods/402e298e-b8a6-4cef-8fa4-a5e667a33ca4/volumes" Oct 31 13:46:04.469904 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:04.469867 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" podUID="df4f4e60-7bc8-4d83-a6eb-97812b694793" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:06.465295 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.465228 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" podUID="8acd1643-d958-467c-8055-de8d1fff7ef2" containerName="main" containerID="cri-o://970af36dd9ee84e8a39df9a594b3f451e6f5c5afc0159c9e3cb091d610a0724e" gracePeriod=45 Oct 31 13:46:06.628022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.627983 5532 generic.go:356] "Generic (PLEG): container finished" podID="8acd1643-d958-467c-8055-de8d1fff7ef2" containerID="970af36dd9ee84e8a39df9a594b3f451e6f5c5afc0159c9e3cb091d610a0724e" exitCode=0 Oct 31 13:46:06.628262 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.628190 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" event={"ID":"8acd1643-d958-467c-8055-de8d1fff7ef2","Type":"ContainerDied","Data":"970af36dd9ee84e8a39df9a594b3f451e6f5c5afc0159c9e3cb091d610a0724e"} Oct 31 13:46:06.810401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.810370 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:46:06.905360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.905326 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-dshm\") pod \"8acd1643-d958-467c-8055-de8d1fff7ef2\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " Oct 31 13:46:06.905642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.905375 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8acd1643-d958-467c-8055-de8d1fff7ef2-kube-api-access-xkt4x\" (UniqueName: \"kubernetes.io/projected/8acd1643-d958-467c-8055-de8d1fff7ef2-kube-api-access-xkt4x\") pod \"8acd1643-d958-467c-8055-de8d1fff7ef2\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " Oct 31 13:46:06.905642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.905430 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-model-cache\") pod \"8acd1643-d958-467c-8055-de8d1fff7ef2\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " Oct 31 13:46:06.905642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.905460 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-home\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-home\") pod \"8acd1643-d958-467c-8055-de8d1fff7ef2\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " Oct 31 13:46:06.905642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.905509 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-tmp-dir\") pod \"8acd1643-d958-467c-8055-de8d1fff7ef2\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " Oct 31 13:46:06.905904 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.905789 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8acd1643-d958-467c-8055-de8d1fff7ef2-tls-certs\" (UniqueName: \"kubernetes.io/secret/8acd1643-d958-467c-8055-de8d1fff7ef2-tls-certs\") pod \"8acd1643-d958-467c-8055-de8d1fff7ef2\" (UID: \"8acd1643-d958-467c-8055-de8d1fff7ef2\") " Oct 31 13:46:06.905968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.905919 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-model-cache" pod "8acd1643-d958-467c-8055-de8d1fff7ef2" (UID: "8acd1643-d958-467c-8055-de8d1fff7ef2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:06.906058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.906028 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-home" pod "8acd1643-d958-467c-8055-de8d1fff7ef2" (UID: "8acd1643-d958-467c-8055-de8d1fff7ef2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:06.906206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.906146 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-tmp-dir" pod "8acd1643-d958-467c-8055-de8d1fff7ef2" (UID: "8acd1643-d958-467c-8055-de8d1fff7ef2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:06.906587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.906563 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:06.906587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.906609 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:06.906587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.906625 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:06.908512 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.908485 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acd1643-d958-467c-8055-de8d1fff7ef2-kube-api-access-xkt4x" pod "8acd1643-d958-467c-8055-de8d1fff7ef2" (UID: "8acd1643-d958-467c-8055-de8d1fff7ef2"). InnerVolumeSpecName "kube-api-access-xkt4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:06.908814 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.908793 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-dshm" pod "8acd1643-d958-467c-8055-de8d1fff7ef2" (UID: "8acd1643-d958-467c-8055-de8d1fff7ef2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:06.909017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:06.908994 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8acd1643-d958-467c-8055-de8d1fff7ef2-tls-certs" pod "8acd1643-d958-467c-8055-de8d1fff7ef2" (UID: "8acd1643-d958-467c-8055-de8d1fff7ef2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:07.008615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:07.008572 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8acd1643-d958-467c-8055-de8d1fff7ef2-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:07.008790 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:07.008627 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8acd1643-d958-467c-8055-de8d1fff7ef2-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:07.008790 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:07.008641 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkt4x\" (UniqueName: \"kubernetes.io/projected/8acd1643-d958-467c-8055-de8d1fff7ef2-kube-api-access-xkt4x\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:07.636957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:07.636923 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" Oct 31 13:46:07.637376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:07.636958 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz" event={"ID":"8acd1643-d958-467c-8055-de8d1fff7ef2","Type":"ContainerDied","Data":"49bbe8a732d27cbeaf195f45db7c835a81dca7b0cc8d762cc383052431f2e558"} Oct 31 13:46:07.637376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:07.637012 5532 scope.go:122] "RemoveContainer" containerID="970af36dd9ee84e8a39df9a594b3f451e6f5c5afc0159c9e3cb091d610a0724e" Oct 31 13:46:07.666780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:07.666755 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz"] Oct 31 13:46:07.673406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:07.673379 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-4fde10/test-model-4fde10-kserve-65cc657c67-7f8kz"] Oct 31 13:46:08.008880 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:08.008836 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8acd1643-d958-467c-8055-de8d1fff7ef2" path="/var/lib/kubelet/pods/8acd1643-d958-467c-8055-de8d1fff7ef2/volumes" Oct 31 13:46:10.370750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:10.370712 5532 patch_prober.go:29] interesting pod/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:46:10.371279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:10.370765 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:46:11.727447 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:11.727413 5532 generic.go:356] "Generic (PLEG): container finished" podID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerID="c3432d0d9aa6508df3debcab6523d69add315abc4d78c18e7e715c63ebe92522" exitCode=0 Oct 31 13:46:11.727974 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:11.727527 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" event={"ID":"60eb6552-2f31-4bd4-bfd7-9286e7620151","Type":"ContainerDied","Data":"c3432d0d9aa6508df3debcab6523d69add315abc4d78c18e7e715c63ebe92522"} Oct 31 13:46:11.728086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:11.728072 5532 scope.go:122] "RemoveContainer" containerID="c3432d0d9aa6508df3debcab6523d69add315abc4d78c18e7e715c63ebe92522" Oct 31 13:46:11.730324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:11.730292 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:46:11.730467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:11.730376 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:46:11.730467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:11.730417 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:46:12.363246 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:12.363211 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh"] Oct 31 13:46:12.367950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:12.367904 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" podUID="f6165f0c-e157-4e98-9317-f0113e333f2e" containerName="maas-api" containerID="cri-o://bc71d2a49a948d1a6286ae7591611a0d9979582ee25f6214501c794b38afab63" gracePeriod=30 Oct 31 13:46:12.896302 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:12.893407 5532 generic.go:356] "Generic (PLEG): container finished" podID="f6165f0c-e157-4e98-9317-f0113e333f2e" containerID="bc71d2a49a948d1a6286ae7591611a0d9979582ee25f6214501c794b38afab63" exitCode=0 Oct 31 13:46:12.897888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:12.897851 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" event={"ID":"f6165f0c-e157-4e98-9317-f0113e333f2e","Type":"ContainerDied","Data":"bc71d2a49a948d1a6286ae7591611a0d9979582ee25f6214501c794b38afab63"} Oct 31 13:46:12.905376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:12.905336 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" event={"ID":"60eb6552-2f31-4bd4-bfd7-9286e7620151","Type":"ContainerStarted","Data":"aa7d56d5bddbe82f00fbb10b4cc4609b710e4d4835da81b8b6bb7877be858444"} Oct 31 13:46:12.984921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:12.984890 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:46:13.129413 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.129379 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-metrics-certs\") pod \"f6165f0c-e157-4e98-9317-f0113e333f2e\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " Oct 31 13:46:13.129619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.129432 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6165f0c-e157-4e98-9317-f0113e333f2e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6165f0c-e157-4e98-9317-f0113e333f2e-openshift-service-ca\") pod \"f6165f0c-e157-4e98-9317-f0113e333f2e\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " Oct 31 13:46:13.129672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.129652 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-maas-api-tls\") pod \"f6165f0c-e157-4e98-9317-f0113e333f2e\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " Oct 31 13:46:13.129742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.129730 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6165f0c-e157-4e98-9317-f0113e333f2e-kube-api-access-fh4st\" (UniqueName: \"kubernetes.io/projected/f6165f0c-e157-4e98-9317-f0113e333f2e-kube-api-access-fh4st\") pod \"f6165f0c-e157-4e98-9317-f0113e333f2e\" (UID: \"f6165f0c-e157-4e98-9317-f0113e333f2e\") " Oct 31 13:46:13.129866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.129839 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6165f0c-e157-4e98-9317-f0113e333f2e-openshift-service-ca" pod "f6165f0c-e157-4e98-9317-f0113e333f2e" (UID: "f6165f0c-e157-4e98-9317-f0113e333f2e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:13.131048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.131020 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6165f0c-e157-4e98-9317-f0113e333f2e-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:13.131912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.131884 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-metrics-certs" pod "f6165f0c-e157-4e98-9317-f0113e333f2e" (UID: "f6165f0c-e157-4e98-9317-f0113e333f2e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:13.132002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.131947 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-maas-api-tls" pod "f6165f0c-e157-4e98-9317-f0113e333f2e" (UID: "f6165f0c-e157-4e98-9317-f0113e333f2e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:13.132070 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.132044 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6165f0c-e157-4e98-9317-f0113e333f2e-kube-api-access-fh4st" pod "f6165f0c-e157-4e98-9317-f0113e333f2e" (UID: "f6165f0c-e157-4e98-9317-f0113e333f2e"). InnerVolumeSpecName "kube-api-access-fh4st". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:13.234389 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.233899 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh4st\" (UniqueName: \"kubernetes.io/projected/f6165f0c-e157-4e98-9317-f0113e333f2e-kube-api-access-fh4st\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:13.234389 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.233942 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:13.234389 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.233957 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f6165f0c-e157-4e98-9317-f0113e333f2e-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:13.663705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.660950 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:46:13.663705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.661109 5532 patch_prober.go:29] interesting pod/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 31 13:46:13.663705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.661154 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 31 13:46:13.915339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.915249 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" event={"ID":"f6165f0c-e157-4e98-9317-f0113e333f2e","Type":"ContainerDied","Data":"6208eee197ffeffa02a5f13ce355468dc618c30ab40e1f04c67f4a5f24196238"} Oct 31 13:46:13.915339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.915322 5532 scope.go:122] "RemoveContainer" containerID="bc71d2a49a948d1a6286ae7591611a0d9979582ee25f6214501c794b38afab63" Oct 31 13:46:13.915918 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.915256 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh" Oct 31 13:46:13.970054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.970018 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh"] Oct 31 13:46:13.974245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:13.974224 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-61f295-69ccb756-655hh"] Oct 31 13:46:14.006378 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:14.006348 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6165f0c-e157-4e98-9317-f0113e333f2e" path="/var/lib/kubelet/pods/f6165f0c-e157-4e98-9317-f0113e333f2e/volumes" Oct 31 13:46:14.468583 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:14.468546 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" podUID="df4f4e60-7bc8-4d83-a6eb-97812b694793" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:14.663735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:14.663694 5532 patch_prober.go:29] interesting pod/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 31 13:46:14.664056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:14.664011 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 31 13:46:15.478627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.478575 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7"] Oct 31 13:46:15.479471 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479455 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f6165f0c-e157-4e98-9317-f0113e333f2e" containerName="maas-api" Oct 31 13:46:15.479519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479471 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6165f0c-e157-4e98-9317-f0113e333f2e" containerName="maas-api" Oct 31 13:46:15.479519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479485 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d9d69e89-8948-4a02-a8c5-35f1466b7a3f" containerName="main" Oct 31 13:46:15.479519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479491 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9d69e89-8948-4a02-a8c5-35f1466b7a3f" containerName="main" Oct 31 13:46:15.479519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479509 5532 cpu_manager.go:414] "containerMap: removing container" podUID="8acd1643-d958-467c-8055-de8d1fff7ef2" containerName="main" Oct 31 13:46:15.479519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479515 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8acd1643-d958-467c-8055-de8d1fff7ef2" containerName="main" Oct 31 13:46:15.479736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479541 5532 cpu_manager.go:414] "containerMap: removing container" podUID="1629727f-2f6f-4d20-a9ae-52ad396a7d65" containerName="revoke-keys" Oct 31 13:46:15.479736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479547 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1629727f-2f6f-4d20-a9ae-52ad396a7d65" containerName="revoke-keys" Oct 31 13:46:15.479736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479569 5532 cpu_manager.go:414] "containerMap: removing container" podUID="402e298e-b8a6-4cef-8fa4-a5e667a33ca4" containerName="payload-processing" Oct 31 13:46:15.479736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479574 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402e298e-b8a6-4cef-8fa4-a5e667a33ca4" containerName="payload-processing" Oct 31 13:46:15.479736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479702 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="1629727f-2f6f-4d20-a9ae-52ad396a7d65" containerName="revoke-keys" Oct 31 13:46:15.479736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479717 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="402e298e-b8a6-4cef-8fa4-a5e667a33ca4" containerName="payload-processing" Oct 31 13:46:15.479736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479728 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6165f0c-e157-4e98-9317-f0113e333f2e" containerName="maas-api" Oct 31 13:46:15.479736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479735 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9d69e89-8948-4a02-a8c5-35f1466b7a3f" containerName="main" Oct 31 13:46:15.479980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.479742 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="8acd1643-d958-467c-8055-de8d1fff7ef2" containerName="main" Oct 31 13:46:15.485026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.485001 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" Oct 31 13:46:15.492464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.492439 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7"] Oct 31 13:46:15.522454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.522422 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:46:15.522649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.522511 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:46:15.605764 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.605733 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7\" (UID: \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" Oct 31 13:46:15.605941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.605901 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7\" (UID: \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" Oct 31 13:46:15.605941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.605930 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vhhj\" (UniqueName: \"kubernetes.io/projected/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-kube-api-access-4vhhj\") pod \"maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7\" (UID: \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" Oct 31 13:46:15.652007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.651969 5532 patch_prober.go:29] interesting pod/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 31 13:46:15.652281 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.652251 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 31 13:46:15.707574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.707545 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7\" (UID: \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" Oct 31 13:46:15.707863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.707731 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7\" (UID: \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" Oct 31 13:46:15.707863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.707762 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vhhj\" (UniqueName: \"kubernetes.io/projected/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-kube-api-access-4vhhj\") pod \"maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7\" (UID: \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" Oct 31 13:46:15.708444 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.708407 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7\" (UID: \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" Oct 31 13:46:15.710101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.710075 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7\" (UID: \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" Oct 31 13:46:15.723516 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.723491 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vhhj\" (UniqueName: \"kubernetes.io/projected/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-kube-api-access-4vhhj\") pod \"maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7\" (UID: \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" Oct 31 13:46:15.808399 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:15.808292 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" Oct 31 13:46:16.213811 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:16.213468 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7"] Oct 31 13:46:16.215204 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:46:16.215170 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01523cc6_f28c_4dc6_9a40_b5a7789baa3a.slice/crio-bcfba5ef763bb7881da2682c74f5a7acf4cb7ddb9bb3a6250dc24ea7b76c94bd WatchSource:0}: Error finding container bcfba5ef763bb7881da2682c74f5a7acf4cb7ddb9bb3a6250dc24ea7b76c94bd: Status 404 returned error can't find the container with id bcfba5ef763bb7881da2682c74f5a7acf4cb7ddb9bb3a6250dc24ea7b76c94bd Oct 31 13:46:16.662895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:16.657691 5532 patch_prober.go:29] interesting pod/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 31 13:46:16.662895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:16.657750 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 31 13:46:16.981046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:16.981009 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" event={"ID":"01523cc6-f28c-4dc6-9a40-b5a7789baa3a","Type":"ContainerStarted","Data":"0dac29201a1cf7747039682b256b93147257f30f7f6c6f27e7e203fe5f0a564a"} Oct 31 13:46:16.981248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:16.981096 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" event={"ID":"01523cc6-f28c-4dc6-9a40-b5a7789baa3a","Type":"ContainerStarted","Data":"bcfba5ef763bb7881da2682c74f5a7acf4cb7ddb9bb3a6250dc24ea7b76c94bd"} Oct 31 13:46:17.003889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:17.003831 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" podStartSLOduration=2.003815304 podStartE2EDuration="2.003815304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:46:17.001175502 +0000 UTC m=+2545.146919354" watchObservedRunningTime="2026-10-31 13:46:17.003815304 +0000 UTC m=+2545.149559133" Oct 31 13:46:17.665625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:17.664264 5532 patch_prober.go:29] interesting pod/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" start-of-body= Oct 31 13:46:17.665625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:17.664339 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.190:15021/healthz/ready\": dial tcp 10.132.0.190:15021: connect: connection refused" Oct 31 13:46:17.991108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:17.990709 5532 generic.go:356] "Generic (PLEG): container finished" podID="01523cc6-f28c-4dc6-9a40-b5a7789baa3a" containerID="0dac29201a1cf7747039682b256b93147257f30f7f6c6f27e7e203fe5f0a564a" exitCode=0 Oct 31 13:46:17.991108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:17.991042 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" event={"ID":"01523cc6-f28c-4dc6-9a40-b5a7789baa3a","Type":"ContainerDied","Data":"0dac29201a1cf7747039682b256b93147257f30f7f6c6f27e7e203fe5f0a564a"} Oct 31 13:46:18.716397 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:18.716366 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:46:18.716970 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:18.716850 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:46:18.717664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:18.717640 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:46:19.168089 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:19.168066 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" Oct 31 13:46:19.206327 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:19.206295 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-openshift-ca\") pod \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\" (UID: \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\") " Oct 31 13:46:19.206478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:19.206392 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-kube-api-access-4vhhj\" (UniqueName: \"kubernetes.io/projected/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-kube-api-access-4vhhj\") pod \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\" (UID: \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\") " Oct 31 13:46:19.206539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:19.206513 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-xks-ca\") pod \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\" (UID: \"01523cc6-f28c-4dc6-9a40-b5a7789baa3a\") " Oct 31 13:46:19.206698 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:19.206668 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-openshift-ca" pod "01523cc6-f28c-4dc6-9a40-b5a7789baa3a" (UID: "01523cc6-f28c-4dc6-9a40-b5a7789baa3a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:19.207464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:19.207440 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:19.208442 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:19.208422 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-xks-ca" pod "01523cc6-f28c-4dc6-9a40-b5a7789baa3a" (UID: "01523cc6-f28c-4dc6-9a40-b5a7789baa3a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:19.208543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:19.208525 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-kube-api-access-4vhhj" pod "01523cc6-f28c-4dc6-9a40-b5a7789baa3a" (UID: "01523cc6-f28c-4dc6-9a40-b5a7789baa3a"). InnerVolumeSpecName "kube-api-access-4vhhj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:19.308805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:19.308724 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vhhj\" (UniqueName: \"kubernetes.io/projected/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-kube-api-access-4vhhj\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:19.308805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:19.308757 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01523cc6-f28c-4dc6-9a40-b5a7789baa3a-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:20.011214 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:20.011182 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" Oct 31 13:46:20.011732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:20.011213 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7" event={"ID":"01523cc6-f28c-4dc6-9a40-b5a7789baa3a","Type":"ContainerDied","Data":"bcfba5ef763bb7881da2682c74f5a7acf4cb7ddb9bb3a6250dc24ea7b76c94bd"} Oct 31 13:46:20.011732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:20.011255 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcfba5ef763bb7881da2682c74f5a7acf4cb7ddb9bb3a6250dc24ea7b76c94bd" Oct 31 13:46:20.803112 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:20.803050 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" Oct 31 13:46:20.951446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:20.951414 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df4f4e60-7bc8-4d83-a6eb-97812b694793-kube-api-access-245qz\" (UniqueName: \"kubernetes.io/projected/df4f4e60-7bc8-4d83-a6eb-97812b694793-kube-api-access-245qz\") pod \"df4f4e60-7bc8-4d83-a6eb-97812b694793\" (UID: \"df4f4e60-7bc8-4d83-a6eb-97812b694793\") " Oct 31 13:46:20.951770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:20.951751 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df4f4e60-7bc8-4d83-a6eb-97812b694793-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4f4e60-7bc8-4d83-a6eb-97812b694793-plugins-config-volume\") pod \"df4f4e60-7bc8-4d83-a6eb-97812b694793\" (UID: \"df4f4e60-7bc8-4d83-a6eb-97812b694793\") " Oct 31 13:46:20.952116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:20.952093 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df4f4e60-7bc8-4d83-a6eb-97812b694793-plugins-config-volume" pod "df4f4e60-7bc8-4d83-a6eb-97812b694793" (UID: "df4f4e60-7bc8-4d83-a6eb-97812b694793"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:20.952322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:20.952308 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df4f4e60-7bc8-4d83-a6eb-97812b694793-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:20.953551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:20.953522 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df4f4e60-7bc8-4d83-a6eb-97812b694793-kube-api-access-245qz" pod "df4f4e60-7bc8-4d83-a6eb-97812b694793" (UID: "df4f4e60-7bc8-4d83-a6eb-97812b694793"). InnerVolumeSpecName "kube-api-access-245qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:21.021291 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:21.021203 5532 generic.go:356] "Generic (PLEG): container finished" podID="df4f4e60-7bc8-4d83-a6eb-97812b694793" containerID="3236bb00f7e74a4b265cfacbd0f9ea3b2a71c52e143145e0843ab00f8bbc0619" exitCode=0 Oct 31 13:46:21.021291 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:21.021247 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" Oct 31 13:46:21.021774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:21.021290 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" event={"ID":"df4f4e60-7bc8-4d83-a6eb-97812b694793","Type":"ContainerDied","Data":"3236bb00f7e74a4b265cfacbd0f9ea3b2a71c52e143145e0843ab00f8bbc0619"} Oct 31 13:46:21.021774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:21.021345 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs" event={"ID":"df4f4e60-7bc8-4d83-a6eb-97812b694793","Type":"ContainerDied","Data":"2c76fd2aea568965cc498c87bbdb483a70518c2ca18118d458dd9c7517d38b5c"} Oct 31 13:46:21.021774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:21.021373 5532 scope.go:122] "RemoveContainer" containerID="3236bb00f7e74a4b265cfacbd0f9ea3b2a71c52e143145e0843ab00f8bbc0619" Oct 31 13:46:21.032918 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:21.032894 5532 scope.go:122] "RemoveContainer" containerID="3236bb00f7e74a4b265cfacbd0f9ea3b2a71c52e143145e0843ab00f8bbc0619" Oct 31 13:46:21.033217 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:46:21.033199 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3236bb00f7e74a4b265cfacbd0f9ea3b2a71c52e143145e0843ab00f8bbc0619\": container with ID starting with 3236bb00f7e74a4b265cfacbd0f9ea3b2a71c52e143145e0843ab00f8bbc0619 not found: ID does not exist" containerID="3236bb00f7e74a4b265cfacbd0f9ea3b2a71c52e143145e0843ab00f8bbc0619" Oct 31 13:46:21.033263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:21.033227 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3236bb00f7e74a4b265cfacbd0f9ea3b2a71c52e143145e0843ab00f8bbc0619"} err="failed to get container status \"3236bb00f7e74a4b265cfacbd0f9ea3b2a71c52e143145e0843ab00f8bbc0619\": rpc error: code = NotFound desc = could not find container \"3236bb00f7e74a4b265cfacbd0f9ea3b2a71c52e143145e0843ab00f8bbc0619\": container with ID starting with 3236bb00f7e74a4b265cfacbd0f9ea3b2a71c52e143145e0843ab00f8bbc0619 not found: ID does not exist" Oct 31 13:46:21.054161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:21.054124 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-245qz\" (UniqueName: \"kubernetes.io/projected/df4f4e60-7bc8-4d83-a6eb-97812b694793-kube-api-access-245qz\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:21.054679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:21.054654 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs"] Oct 31 13:46:21.066101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:21.066078 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-61f295-865f8d99c5-zjwcs"] Oct 31 13:46:22.005985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:22.005953 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df4f4e60-7bc8-4d83-a6eb-97812b694793" path="/var/lib/kubelet/pods/df4f4e60-7bc8-4d83-a6eb-97812b694793/volumes" Oct 31 13:46:25.553005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:25.552969 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm"] Oct 31 13:46:25.562605 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:25.562561 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4232c-c5gnm"] Oct 31 13:46:25.773156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:25.773117 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb"] Oct 31 13:46:25.792456 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:25.792420 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf"] Oct 31 13:46:26.006172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:26.006136 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a213358-40d2-4ded-9697-d6b2739fc99b" path="/var/lib/kubelet/pods/4a213358-40d2-4ded-9697-d6b2739fc99b/volumes" Oct 31 13:46:26.651297 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:26.651262 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68"] Oct 31 13:46:26.651771 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:26.651548 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" podUID="38485377-d08e-459c-a544-50cf06cfe529" containerName="maas-api" containerID="cri-o://1fe824733ecd7701b3fbd21a000b4c540d190cd61d6dd8705d4d2586214c0dd5" gracePeriod=30 Oct 31 13:46:27.170338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.170310 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:46:27.175367 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.175328 5532 generic.go:356] "Generic (PLEG): container finished" podID="38485377-d08e-459c-a544-50cf06cfe529" containerID="1fe824733ecd7701b3fbd21a000b4c540d190cd61d6dd8705d4d2586214c0dd5" exitCode=0 Oct 31 13:46:27.175367 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.175386 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" event={"ID":"38485377-d08e-459c-a544-50cf06cfe529","Type":"ContainerDied","Data":"1fe824733ecd7701b3fbd21a000b4c540d190cd61d6dd8705d4d2586214c0dd5"} Oct 31 13:46:27.175846 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.175429 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" Oct 31 13:46:27.175846 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.175467 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68" event={"ID":"38485377-d08e-459c-a544-50cf06cfe529","Type":"ContainerDied","Data":"5cbfc6fc5ff06385a2a9de93679f7f8d9f9e5fed2783cea7c3151fd39a677d58"} Oct 31 13:46:27.175846 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.175496 5532 scope.go:122] "RemoveContainer" containerID="1fe824733ecd7701b3fbd21a000b4c540d190cd61d6dd8705d4d2586214c0dd5" Oct 31 13:46:27.186722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.186704 5532 scope.go:122] "RemoveContainer" containerID="1fe824733ecd7701b3fbd21a000b4c540d190cd61d6dd8705d4d2586214c0dd5" Oct 31 13:46:27.186976 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:46:27.186954 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fe824733ecd7701b3fbd21a000b4c540d190cd61d6dd8705d4d2586214c0dd5\": container with ID starting with 1fe824733ecd7701b3fbd21a000b4c540d190cd61d6dd8705d4d2586214c0dd5 not found: ID does not exist" containerID="1fe824733ecd7701b3fbd21a000b4c540d190cd61d6dd8705d4d2586214c0dd5" Oct 31 13:46:27.187034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.186985 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fe824733ecd7701b3fbd21a000b4c540d190cd61d6dd8705d4d2586214c0dd5"} err="failed to get container status \"1fe824733ecd7701b3fbd21a000b4c540d190cd61d6dd8705d4d2586214c0dd5\": rpc error: code = NotFound desc = could not find container \"1fe824733ecd7701b3fbd21a000b4c540d190cd61d6dd8705d4d2586214c0dd5\": container with ID starting with 1fe824733ecd7701b3fbd21a000b4c540d190cd61d6dd8705d4d2586214c0dd5 not found: ID does not exist" Oct 31 13:46:27.307857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.307827 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38485377-d08e-459c-a544-50cf06cfe529-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38485377-d08e-459c-a544-50cf06cfe529-openshift-service-ca\") pod \"38485377-d08e-459c-a544-50cf06cfe529\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " Oct 31 13:46:27.308048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.307880 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-maas-api-tls\") pod \"38485377-d08e-459c-a544-50cf06cfe529\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " Oct 31 13:46:27.308048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.307967 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38485377-d08e-459c-a544-50cf06cfe529-kube-api-access-cqd58\" (UniqueName: \"kubernetes.io/projected/38485377-d08e-459c-a544-50cf06cfe529-kube-api-access-cqd58\") pod \"38485377-d08e-459c-a544-50cf06cfe529\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " Oct 31 13:46:27.308048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.308042 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-metrics-certs\" (UniqueName: \"kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-metrics-certs\") pod \"38485377-d08e-459c-a544-50cf06cfe529\" (UID: \"38485377-d08e-459c-a544-50cf06cfe529\") " Oct 31 13:46:27.308239 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.308210 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38485377-d08e-459c-a544-50cf06cfe529-openshift-service-ca" pod "38485377-d08e-459c-a544-50cf06cfe529" (UID: "38485377-d08e-459c-a544-50cf06cfe529"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:27.308725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.308704 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/38485377-d08e-459c-a544-50cf06cfe529-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:27.310254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.310226 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-metrics-certs" pod "38485377-d08e-459c-a544-50cf06cfe529" (UID: "38485377-d08e-459c-a544-50cf06cfe529"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:27.310348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.310304 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-maas-api-tls" pod "38485377-d08e-459c-a544-50cf06cfe529" (UID: "38485377-d08e-459c-a544-50cf06cfe529"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:27.310397 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.310350 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38485377-d08e-459c-a544-50cf06cfe529-kube-api-access-cqd58" pod "38485377-d08e-459c-a544-50cf06cfe529" (UID: "38485377-d08e-459c-a544-50cf06cfe529"). InnerVolumeSpecName "kube-api-access-cqd58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:27.364791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.364704 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h"] Oct 31 13:46:27.369415 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.369389 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-61da9d-z4g9h"] Oct 31 13:46:27.410148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.410117 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:27.410321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.410154 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqd58\" (UniqueName: \"kubernetes.io/projected/38485377-d08e-459c-a544-50cf06cfe529-kube-api-access-cqd58\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:27.410321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.410200 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/38485377-d08e-459c-a544-50cf06cfe529-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:27.536402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.536372 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68"] Oct 31 13:46:27.540416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:27.540392 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-61da9d-77fb898559-tlv68"] Oct 31 13:46:28.006145 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:28.006114 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1629727f-2f6f-4d20-a9ae-52ad396a7d65" path="/var/lib/kubelet/pods/1629727f-2f6f-4d20-a9ae-52ad396a7d65/volumes" Oct 31 13:46:28.006667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:28.006652 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38485377-d08e-459c-a544-50cf06cfe529" path="/var/lib/kubelet/pods/38485377-d08e-459c-a544-50cf06cfe529/volumes" Oct 31 13:46:30.774430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:30.774385 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" podUID="ad4ab732-0660-42a4-80e8-ebed759dd941" containerName="payload-processing" containerID="cri-o://dfdc707f45b16b8d3e5df23baae013723448881a433a8e0242248cdc79613eef" gracePeriod=25 Oct 31 13:46:30.792981 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:30.792925 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" podUID="f61f37d8-8b15-4955-bf9d-33a85b280110" containerName="payload-pre-processing" containerID="cri-o://77def19e6fb0460e6d6f6b79ede5dd3230e36d81693c041bde51d99e77f99b54" gracePeriod=25 Oct 31 13:46:31.762938 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:46:31.762899 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:46:33.744969 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:33.744926 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr"] Oct 31 13:46:33.745501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:33.745199 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" podUID="3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" containerName="istio-proxy" containerID="cri-o://383c0bee28290db60b1efb038597fd9c398e22225df37ef48c6ac21dc379fb41" gracePeriod=30 Oct 31 13:46:33.751468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:33.751412 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn"] Oct 31 13:46:33.751678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:33.751654 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" podUID="e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" containerName="istio-proxy" containerID="cri-o://accaf05a1efdd5fae8ed7ee16e88c0a011378cbe77b050d2069f3811a8a8fcc2" gracePeriod=30 Oct 31 13:46:34.253425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.253392 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67"] Oct 31 13:46:34.254195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.254179 5532 cpu_manager.go:414] "containerMap: removing container" podUID="01523cc6-f28c-4dc6-9a40-b5a7789baa3a" containerName="revoke-keys" Oct 31 13:46:34.254257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.254196 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01523cc6-f28c-4dc6-9a40-b5a7789baa3a" containerName="revoke-keys" Oct 31 13:46:34.254257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.254223 5532 cpu_manager.go:414] "containerMap: removing container" podUID="df4f4e60-7bc8-4d83-a6eb-97812b694793" containerName="payload-pre-processing" Oct 31 13:46:34.254257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.254230 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df4f4e60-7bc8-4d83-a6eb-97812b694793" containerName="payload-pre-processing" Oct 31 13:46:34.254257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.254247 5532 cpu_manager.go:414] "containerMap: removing container" podUID="38485377-d08e-459c-a544-50cf06cfe529" containerName="maas-api" Oct 31 13:46:34.254257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.254252 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38485377-d08e-459c-a544-50cf06cfe529" containerName="maas-api" Oct 31 13:46:34.254525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.254394 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="01523cc6-f28c-4dc6-9a40-b5a7789baa3a" containerName="revoke-keys" Oct 31 13:46:34.254525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.254405 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="df4f4e60-7bc8-4d83-a6eb-97812b694793" containerName="payload-pre-processing" Oct 31 13:46:34.254525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.254415 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="38485377-d08e-459c-a544-50cf06cfe529" containerName="maas-api" Oct 31 13:46:34.259996 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.259977 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" Oct 31 13:46:34.266487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.266461 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:46:34.266730 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.266712 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:46:34.269896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.269873 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67"] Oct 31 13:46:34.370667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.370634 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67\" (UID: \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" Oct 31 13:46:34.370822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.370698 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67\" (UID: \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" Oct 31 13:46:34.370918 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.370904 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj5nt\" (UniqueName: \"kubernetes.io/projected/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-kube-api-access-sj5nt\") pod \"maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67\" (UID: \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" Oct 31 13:46:34.472429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.472390 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67\" (UID: \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" Oct 31 13:46:34.472587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.472469 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67\" (UID: \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" Oct 31 13:46:34.472722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.472705 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj5nt\" (UniqueName: \"kubernetes.io/projected/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-kube-api-access-sj5nt\") pod \"maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67\" (UID: \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" Oct 31 13:46:34.473205 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.473176 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67\" (UID: \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" Oct 31 13:46:34.475026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.475003 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67\" (UID: \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" Oct 31 13:46:34.486728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.486705 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj5nt\" (UniqueName: \"kubernetes.io/projected/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-kube-api-access-sj5nt\") pod \"maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67\" (UID: \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" Oct 31 13:46:34.577009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.576930 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" Oct 31 13:46:34.882590 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:46:34.882558 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a26255b_8277_4bbd_95e9_20afdc2ec6ea.slice/crio-c6e33253ecf6ec6af7df6be2dfb06c23726a2155e7952e40b6c9cfc42cfa9774 WatchSource:0}: Error finding container c6e33253ecf6ec6af7df6be2dfb06c23726a2155e7952e40b6c9cfc42cfa9774: Status 404 returned error can't find the container with id c6e33253ecf6ec6af7df6be2dfb06c23726a2155e7952e40b6c9cfc42cfa9774 Oct 31 13:46:34.883169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:34.883145 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67"] Oct 31 13:46:35.351632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:35.345480 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" event={"ID":"8a26255b-8277-4bbd-95e9-20afdc2ec6ea","Type":"ContainerStarted","Data":"8ec376f9dd5a039996cafa559ee13d45705dbe32523aab05fdcf2dc5a1562157"} Oct 31 13:46:35.351632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:35.345549 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" event={"ID":"8a26255b-8277-4bbd-95e9-20afdc2ec6ea","Type":"ContainerStarted","Data":"c6e33253ecf6ec6af7df6be2dfb06c23726a2155e7952e40b6c9cfc42cfa9774"} Oct 31 13:46:35.385616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:35.382649 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" podStartSLOduration=1.382630395 podStartE2EDuration="1.382630395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:46:35.376725701 +0000 UTC m=+2563.522469530" watchObservedRunningTime="2026-10-31 13:46:35.382630395 +0000 UTC m=+2563.528374228" Oct 31 13:46:35.491999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:35.491909 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" podUID="f61f37d8-8b15-4955-bf9d-33a85b280110" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:35.496427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:35.495574 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" podUID="ad4ab732-0660-42a4-80e8-ebed759dd941" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:36.123584 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.123501 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:46:36.123584 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.123564 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" podUID="3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:46:36.157685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.157649 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl"] Oct 31 13:46:36.158060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.158029 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" podUID="e147e116-ed36-4be3-82ee-b37c85f31665" containerName="istio-proxy" containerID="cri-o://33cf7bb242187c5c0d325c65385fb019605feb924f1271e15e090eca3c6b17a8" gracePeriod=30 Oct 31 13:46:36.165953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.165925 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4"] Oct 31 13:46:36.166207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.166168 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" podUID="27643442-2613-43f3-a046-4a3e8dcd56db" containerName="istio-proxy" containerID="cri-o://fb1ef483cd12f6601133838a5ae08a190b326d6a978a9a4edd08a7f7c1185cf9" gracePeriod=30 Oct 31 13:46:36.171427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.171399 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx"] Oct 31 13:46:36.171827 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.171630 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerName="istio-proxy" containerID="cri-o://5ee26139b61dc69334a980b4c383da3c7af64013ccb75bbe61c57406599e927a" gracePeriod=30 Oct 31 13:46:36.176039 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.176013 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh"] Oct 31 13:46:36.176246 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.176224 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" podUID="d01fddda-bc47-40df-914b-d27a8363654e" containerName="istio-proxy" containerID="cri-o://e33bae3b70cfc93db66d404d8d84380adf900a5f4004acc52bc89f908a0a99b5" gracePeriod=30 Oct 31 13:46:36.180505 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.180479 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r"] Oct 31 13:46:36.180748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.180723 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" podUID="d7327fd4-10e0-41ae-b540-03778f4df456" containerName="istio-proxy" containerID="cri-o://1f4b5034ac08ec62ebde3d24c3813f8bc650dfdec2e70c0309144c1090285a19" gracePeriod=30 Oct 31 13:46:36.187928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.187905 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff"] Oct 31 13:46:36.188075 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.188054 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" podUID="b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" containerName="istio-proxy" containerID="cri-o://447a0378fd40567b30c4e43c7bd10ec38c9f0743b947f30919164e60e01fd5c7" gracePeriod=30 Oct 31 13:46:36.373857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.373277 5532 generic.go:356] "Generic (PLEG): container finished" podID="8a26255b-8277-4bbd-95e9-20afdc2ec6ea" containerID="8ec376f9dd5a039996cafa559ee13d45705dbe32523aab05fdcf2dc5a1562157" exitCode=0 Oct 31 13:46:36.373857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.373626 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" event={"ID":"8a26255b-8277-4bbd-95e9-20afdc2ec6ea","Type":"ContainerDied","Data":"8ec376f9dd5a039996cafa559ee13d45705dbe32523aab05fdcf2dc5a1562157"} Oct 31 13:46:36.682279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.682240 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6"] Oct 31 13:46:36.687401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.687379 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" Oct 31 13:46:36.696107 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.696081 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6"] Oct 31 13:46:36.763814 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.763779 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6\" (UID: \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" Oct 31 13:46:36.764020 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.763836 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpml2\" (UniqueName: \"kubernetes.io/projected/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-kube-api-access-zpml2\") pod \"maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6\" (UID: \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" Oct 31 13:46:36.764020 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.763953 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6\" (UID: \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" Oct 31 13:46:36.865256 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.865210 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6\" (UID: \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" Oct 31 13:46:36.865454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.865290 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpml2\" (UniqueName: \"kubernetes.io/projected/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-kube-api-access-zpml2\") pod \"maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6\" (UID: \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" Oct 31 13:46:36.865454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.865334 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6\" (UID: \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" Oct 31 13:46:36.865963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.865936 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6\" (UID: \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" Oct 31 13:46:36.867722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.867699 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6\" (UID: \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" Oct 31 13:46:36.879737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:36.879714 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpml2\" (UniqueName: \"kubernetes.io/projected/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-kube-api-access-zpml2\") pod \"maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6\" (UID: \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" Oct 31 13:46:37.005393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.005308 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" Oct 31 13:46:37.292602 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.292565 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6"] Oct 31 13:46:37.293169 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:46:37.293146 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cb2933a_c77e_42d9_b3ba_adad4aa5bb14.slice/crio-089508f8670763dd63f3125c7b3413485e91104b2ab0466ccd5a993b500c5ced WatchSource:0}: Error finding container 089508f8670763dd63f3125c7b3413485e91104b2ab0466ccd5a993b500c5ced: Status 404 returned error can't find the container with id 089508f8670763dd63f3125c7b3413485e91104b2ab0466ccd5a993b500c5ced Oct 31 13:46:37.318560 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.318530 5532 patch_prober.go:29] interesting pod/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:46:37.318695 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.318583 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" podUID="e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:46:37.385658 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.385620 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" event={"ID":"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14","Type":"ContainerStarted","Data":"089508f8670763dd63f3125c7b3413485e91104b2ab0466ccd5a993b500c5ced"} Oct 31 13:46:37.539993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.539925 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" Oct 31 13:46:37.679852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.679823 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-kube-api-access-sj5nt\" (UniqueName: \"kubernetes.io/projected/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-kube-api-access-sj5nt\") pod \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\" (UID: \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\") " Oct 31 13:46:37.680005 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.679953 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-xks-ca\") pod \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\" (UID: \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\") " Oct 31 13:46:37.680082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.680064 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-openshift-ca\") pod \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\" (UID: \"8a26255b-8277-4bbd-95e9-20afdc2ec6ea\") " Oct 31 13:46:37.680427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.680402 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-openshift-ca" pod "8a26255b-8277-4bbd-95e9-20afdc2ec6ea" (UID: "8a26255b-8277-4bbd-95e9-20afdc2ec6ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:37.680680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.680662 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:37.681869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.681845 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-kube-api-access-sj5nt" pod "8a26255b-8277-4bbd-95e9-20afdc2ec6ea" (UID: "8a26255b-8277-4bbd-95e9-20afdc2ec6ea"). InnerVolumeSpecName "kube-api-access-sj5nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:37.681959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.681921 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-xks-ca" pod "8a26255b-8277-4bbd-95e9-20afdc2ec6ea" (UID: "8a26255b-8277-4bbd-95e9-20afdc2ec6ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:37.781910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.781876 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj5nt\" (UniqueName: \"kubernetes.io/projected/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-kube-api-access-sj5nt\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:37.781910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:37.781907 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a26255b-8277-4bbd-95e9-20afdc2ec6ea-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:38.399922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:38.399890 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" Oct 31 13:46:38.400408 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:38.400264 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67" event={"ID":"8a26255b-8277-4bbd-95e9-20afdc2ec6ea","Type":"ContainerDied","Data":"c6e33253ecf6ec6af7df6be2dfb06c23726a2155e7952e40b6c9cfc42cfa9774"} Oct 31 13:46:38.400408 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:38.400318 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6e33253ecf6ec6af7df6be2dfb06c23726a2155e7952e40b6c9cfc42cfa9774" Oct 31 13:46:38.402438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:38.402410 5532 generic.go:356] "Generic (PLEG): container finished" podID="7cb2933a-c77e-42d9-b3ba-adad4aa5bb14" containerID="cdb8221749a137a15b3a7509a39ba266e6a43e841512c34a16ae8736afe31bd2" exitCode=0 Oct 31 13:46:38.402580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:38.402437 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" event={"ID":"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14","Type":"ContainerDied","Data":"cdb8221749a137a15b3a7509a39ba266e6a43e841512c34a16ae8736afe31bd2"} Oct 31 13:46:38.497821 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:38.497733 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7"] Oct 31 13:46:38.502788 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:38.502758 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-4fde10-h7nl7"] Oct 31 13:46:39.057347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.057319 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:46:39.130844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.130819 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:46:39.209074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.209040 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-certs\") pod \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " Oct 31 13:46:39.209248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.209125 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istiod-ca-cert\") pod \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " Oct 31 13:46:39.209248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.209162 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-credential-socket\") pod \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " Oct 31 13:46:39.209248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.209226 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-podinfo\") pod \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " Oct 31 13:46:39.209450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.209267 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-token\" (UniqueName: \"kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-token\") pod \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " Oct 31 13:46:39.209748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.209720 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-socket\") pod \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " Oct 31 13:46:39.209748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.209326 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-certs" pod "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" (UID: "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:39.209957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.209573 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-credential-socket" pod "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" (UID: "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:39.209957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.209618 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istiod-ca-cert" pod "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" (UID: "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:39.209957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.209804 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-envoy\") pod \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " Oct 31 13:46:39.209957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.209837 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istiod-ca-cert\") pod \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " Oct 31 13:46:39.209957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.209928 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-socket\") pod \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " Oct 31 13:46:39.210684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.209972 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-envoy\") pod \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " Oct 31 13:46:39.210684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.210199 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-certs\") pod \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " Oct 31 13:46:39.210684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.210231 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-kube-api-access-2gzbc\" (UniqueName: \"kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-kube-api-access-2gzbc\") pod \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " Oct 31 13:46:39.210684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.210253 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-socket" pod "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" (UID: "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:39.210684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.210292 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-credential-socket\") pod \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " Oct 31 13:46:39.210684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.210352 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-data\") pod \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " Oct 31 13:46:39.210684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.210432 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-kube-api-access-n7f5m\" (UniqueName: \"kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-kube-api-access-n7f5m\") pod \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\" (UID: \"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4\") " Oct 31 13:46:39.210684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.210437 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-certs" pod "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" (UID: "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:39.210684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.210474 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-token\" (UniqueName: \"kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-token\") pod \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " Oct 31 13:46:39.210684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.210530 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-data\") pod \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " Oct 31 13:46:39.210684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.210557 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-podinfo\") pod \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\" (UID: \"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6\") " Oct 31 13:46:39.210684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.210643 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-credential-socket" pod "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" (UID: "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:39.211378 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.210987 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istiod-ca-cert" pod "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" (UID: "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:39.212203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.211529 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.212203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.211556 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.212203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.211572 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.212203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.211587 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.212203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.211617 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.212203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.211631 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.212203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.211645 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.213023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.212712 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-data" pod "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" (UID: "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:39.213023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.212786 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-podinfo" pod "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" (UID: "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:39.213023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.212993 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-data" pod "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" (UID: "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:39.213308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.213282 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-socket" pod "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" (UID: "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:39.213961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.213934 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-podinfo" pod "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" (UID: "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:39.214143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.214106 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-token" pod "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" (UID: "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:39.215011 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.214984 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-envoy" pod "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" (UID: "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:39.215666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.215640 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-token" pod "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" (UID: "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:39.215927 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.215902 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-kube-api-access-2gzbc" pod "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" (UID: "e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6"). InnerVolumeSpecName "kube-api-access-2gzbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:39.216422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.216397 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-envoy" pod "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" (UID: "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:39.216843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.216821 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-kube-api-access-n7f5m" pod "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" (UID: "3dde3d07-0294-4137-a7f2-4fd2e4a5feb4"). InnerVolumeSpecName "kube-api-access-n7f5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:39.313406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.313328 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gzbc\" (UniqueName: \"kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-kube-api-access-2gzbc\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.313406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.313360 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.313406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.313374 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7f5m\" (UniqueName: \"kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-kube-api-access-n7f5m\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.313406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.313385 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.313406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.313394 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.313406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.313402 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.313406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.313410 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.313778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.313418 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.313778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.313427 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.313778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.313437 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.313778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.313450 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.413109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.413074 5532 generic.go:356] "Generic (PLEG): container finished" podID="e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" containerID="accaf05a1efdd5fae8ed7ee16e88c0a011378cbe77b050d2069f3811a8a8fcc2" exitCode=0 Oct 31 13:46:39.413528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.413122 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" Oct 31 13:46:39.413528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.413151 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" event={"ID":"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6","Type":"ContainerDied","Data":"accaf05a1efdd5fae8ed7ee16e88c0a011378cbe77b050d2069f3811a8a8fcc2"} Oct 31 13:46:39.413528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.413205 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn" event={"ID":"e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6","Type":"ContainerDied","Data":"75d8f78665d16da2f031877601d889c24a19507d40a46a34606bc946c5d15783"} Oct 31 13:46:39.413528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.413227 5532 scope.go:122] "RemoveContainer" containerID="accaf05a1efdd5fae8ed7ee16e88c0a011378cbe77b050d2069f3811a8a8fcc2" Oct 31 13:46:39.414945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.414921 5532 generic.go:356] "Generic (PLEG): container finished" podID="3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" containerID="383c0bee28290db60b1efb038597fd9c398e22225df37ef48c6ac21dc379fb41" exitCode=0 Oct 31 13:46:39.415056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.414953 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" event={"ID":"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4","Type":"ContainerDied","Data":"383c0bee28290db60b1efb038597fd9c398e22225df37ef48c6ac21dc379fb41"} Oct 31 13:46:39.415056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.414982 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" Oct 31 13:46:39.415056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.415008 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr" event={"ID":"3dde3d07-0294-4137-a7f2-4fd2e4a5feb4","Type":"ContainerDied","Data":"3e95b005030d3a4c53fab2fc86e5e144b47a40e3250ac9919ca9be82f0117f6a"} Oct 31 13:46:39.429126 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.429105 5532 scope.go:122] "RemoveContainer" containerID="accaf05a1efdd5fae8ed7ee16e88c0a011378cbe77b050d2069f3811a8a8fcc2" Oct 31 13:46:39.429428 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:46:39.429406 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"accaf05a1efdd5fae8ed7ee16e88c0a011378cbe77b050d2069f3811a8a8fcc2\": container with ID starting with accaf05a1efdd5fae8ed7ee16e88c0a011378cbe77b050d2069f3811a8a8fcc2 not found: ID does not exist" containerID="accaf05a1efdd5fae8ed7ee16e88c0a011378cbe77b050d2069f3811a8a8fcc2" Oct 31 13:46:39.429503 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.429434 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"accaf05a1efdd5fae8ed7ee16e88c0a011378cbe77b050d2069f3811a8a8fcc2"} err="failed to get container status \"accaf05a1efdd5fae8ed7ee16e88c0a011378cbe77b050d2069f3811a8a8fcc2\": rpc error: code = NotFound desc = could not find container \"accaf05a1efdd5fae8ed7ee16e88c0a011378cbe77b050d2069f3811a8a8fcc2\": container with ID starting with accaf05a1efdd5fae8ed7ee16e88c0a011378cbe77b050d2069f3811a8a8fcc2 not found: ID does not exist" Oct 31 13:46:39.429503 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.429450 5532 scope.go:122] "RemoveContainer" containerID="383c0bee28290db60b1efb038597fd9c398e22225df37ef48c6ac21dc379fb41" Oct 31 13:46:39.442917 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.442722 5532 scope.go:122] "RemoveContainer" containerID="383c0bee28290db60b1efb038597fd9c398e22225df37ef48c6ac21dc379fb41" Oct 31 13:46:39.443232 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:46:39.443203 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"383c0bee28290db60b1efb038597fd9c398e22225df37ef48c6ac21dc379fb41\": container with ID starting with 383c0bee28290db60b1efb038597fd9c398e22225df37ef48c6ac21dc379fb41 not found: ID does not exist" containerID="383c0bee28290db60b1efb038597fd9c398e22225df37ef48c6ac21dc379fb41" Oct 31 13:46:39.443357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.443242 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"383c0bee28290db60b1efb038597fd9c398e22225df37ef48c6ac21dc379fb41"} err="failed to get container status \"383c0bee28290db60b1efb038597fd9c398e22225df37ef48c6ac21dc379fb41\": rpc error: code = NotFound desc = could not find container \"383c0bee28290db60b1efb038597fd9c398e22225df37ef48c6ac21dc379fb41\": container with ID starting with 383c0bee28290db60b1efb038597fd9c398e22225df37ef48c6ac21dc379fb41 not found: ID does not exist" Oct 31 13:46:39.446492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.446471 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn"] Oct 31 13:46:39.450649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.450622 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-4fgqn"] Oct 31 13:46:39.467247 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.467218 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr"] Oct 31 13:46:39.472618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.472582 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f4232c-openshift-default-8664f8f795-pm4nr"] Oct 31 13:46:39.583043 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.583018 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" Oct 31 13:46:39.718626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.718582 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-xks-ca\") pod \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\" (UID: \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\") " Oct 31 13:46:39.718843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.718809 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-openshift-ca\") pod \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\" (UID: \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\") " Oct 31 13:46:39.719047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.719028 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-kube-api-access-zpml2\" (UniqueName: \"kubernetes.io/projected/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-kube-api-access-zpml2\") pod \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\" (UID: \"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14\") " Oct 31 13:46:39.719220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.719200 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-openshift-ca" pod "7cb2933a-c77e-42d9-b3ba-adad4aa5bb14" (UID: "7cb2933a-c77e-42d9-b3ba-adad4aa5bb14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:39.719698 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.719679 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.720548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.720521 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-xks-ca" pod "7cb2933a-c77e-42d9-b3ba-adad4aa5bb14" (UID: "7cb2933a-c77e-42d9-b3ba-adad4aa5bb14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:46:39.721107 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.721082 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-kube-api-access-zpml2" pod "7cb2933a-c77e-42d9-b3ba-adad4aa5bb14" (UID: "7cb2933a-c77e-42d9-b3ba-adad4aa5bb14"). InnerVolumeSpecName "kube-api-access-zpml2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:39.821885 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.821853 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpml2\" (UniqueName: \"kubernetes.io/projected/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-kube-api-access-zpml2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:39.821885 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:39.821883 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:40.007413 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:40.007380 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01523cc6-f28c-4dc6-9a40-b5a7789baa3a" path="/var/lib/kubelet/pods/01523cc6-f28c-4dc6-9a40-b5a7789baa3a/volumes" Oct 31 13:46:40.008017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:40.007999 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" path="/var/lib/kubelet/pods/3dde3d07-0294-4137-a7f2-4fd2e4a5feb4/volumes" Oct 31 13:46:40.008518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:40.008503 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" path="/var/lib/kubelet/pods/e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6/volumes" Oct 31 13:46:40.432499 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:40.432460 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" event={"ID":"7cb2933a-c77e-42d9-b3ba-adad4aa5bb14","Type":"ContainerDied","Data":"089508f8670763dd63f3125c7b3413485e91104b2ab0466ccd5a993b500c5ced"} Oct 31 13:46:40.432499 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:40.432485 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6" Oct 31 13:46:40.433141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:40.432510 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="089508f8670763dd63f3125c7b3413485e91104b2ab0466ccd5a993b500c5ced" Oct 31 13:46:41.219728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.219661 5532 patch_prober.go:29] interesting pod/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 31 13:46:41.219728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.219715 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 31 13:46:41.447797 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.447204 5532 generic.go:356] "Generic (PLEG): container finished" podID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerID="5ee26139b61dc69334a980b4c383da3c7af64013ccb75bbe61c57406599e927a" exitCode=0 Oct 31 13:46:41.447797 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.447361 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" event={"ID":"bfe5d1c7-179a-42e0-8457-3cba517be508","Type":"ContainerDied","Data":"5ee26139b61dc69334a980b4c383da3c7af64013ccb75bbe61c57406599e927a"} Oct 31 13:46:41.449954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.449776 5532 generic.go:356] "Generic (PLEG): container finished" podID="e147e116-ed36-4be3-82ee-b37c85f31665" containerID="33cf7bb242187c5c0d325c65385fb019605feb924f1271e15e090eca3c6b17a8" exitCode=0 Oct 31 13:46:41.449954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.449896 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" event={"ID":"e147e116-ed36-4be3-82ee-b37c85f31665","Type":"ContainerDied","Data":"33cf7bb242187c5c0d325c65385fb019605feb924f1271e15e090eca3c6b17a8"} Oct 31 13:46:41.452936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.452769 5532 generic.go:356] "Generic (PLEG): container finished" podID="27643442-2613-43f3-a046-4a3e8dcd56db" containerID="fb1ef483cd12f6601133838a5ae08a190b326d6a978a9a4edd08a7f7c1185cf9" exitCode=0 Oct 31 13:46:41.452936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.452865 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" event={"ID":"27643442-2613-43f3-a046-4a3e8dcd56db","Type":"ContainerDied","Data":"fb1ef483cd12f6601133838a5ae08a190b326d6a978a9a4edd08a7f7c1185cf9"} Oct 31 13:46:41.455181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.455040 5532 generic.go:356] "Generic (PLEG): container finished" podID="b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" containerID="447a0378fd40567b30c4e43c7bd10ec38c9f0743b947f30919164e60e01fd5c7" exitCode=0 Oct 31 13:46:41.455181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.455134 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" event={"ID":"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663","Type":"ContainerDied","Data":"447a0378fd40567b30c4e43c7bd10ec38c9f0743b947f30919164e60e01fd5c7"} Oct 31 13:46:41.456886 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.456786 5532 generic.go:356] "Generic (PLEG): container finished" podID="d01fddda-bc47-40df-914b-d27a8363654e" containerID="e33bae3b70cfc93db66d404d8d84380adf900a5f4004acc52bc89f908a0a99b5" exitCode=0 Oct 31 13:46:41.456886 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.456839 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" event={"ID":"d01fddda-bc47-40df-914b-d27a8363654e","Type":"ContainerDied","Data":"e33bae3b70cfc93db66d404d8d84380adf900a5f4004acc52bc89f908a0a99b5"} Oct 31 13:46:41.458524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.458465 5532 generic.go:356] "Generic (PLEG): container finished" podID="d7327fd4-10e0-41ae-b540-03778f4df456" containerID="1f4b5034ac08ec62ebde3d24c3813f8bc650dfdec2e70c0309144c1090285a19" exitCode=0 Oct 31 13:46:41.458524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.458492 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" event={"ID":"d7327fd4-10e0-41ae-b540-03778f4df456","Type":"ContainerDied","Data":"1f4b5034ac08ec62ebde3d24c3813f8bc650dfdec2e70c0309144c1090285a19"} Oct 31 13:46:41.624958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.619283 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:46:41.629898 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.629872 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:46:41.646731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.646709 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:46:41.651154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.651134 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:46:41.656933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.656913 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:46:41.660468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.660446 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:46:41.786780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.786696 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-certs\") pod \"bfe5d1c7-179a-42e0-8457-3cba517be508\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " Oct 31 13:46:41.787749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.787720 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-kube-api-access-5srcn\" (UniqueName: \"kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-kube-api-access-5srcn\") pod \"d01fddda-bc47-40df-914b-d27a8363654e\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " Oct 31 13:46:41.787980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.787959 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-socket\") pod \"e147e116-ed36-4be3-82ee-b37c85f31665\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " Oct 31 13:46:41.788157 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.788129 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-istio-token\" (UniqueName: \"kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-istio-token\") pod \"d7327fd4-10e0-41ae-b540-03778f4df456\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " Oct 31 13:46:41.788292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.788178 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-kube-api-access-x5k22\" (UniqueName: \"kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-kube-api-access-x5k22\") pod \"bfe5d1c7-179a-42e0-8457-3cba517be508\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " Oct 31 13:46:41.788292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.788208 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-token\" (UniqueName: \"kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-token\") pod \"bfe5d1c7-179a-42e0-8457-3cba517be508\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " Oct 31 13:46:41.788292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.788251 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d7327fd4-10e0-41ae-b540-03778f4df456-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7327fd4-10e0-41ae-b540-03778f4df456-istio-podinfo\") pod \"d7327fd4-10e0-41ae-b540-03778f4df456\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " Oct 31 13:46:41.788292 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.788280 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27643442-2613-43f3-a046-4a3e8dcd56db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27643442-2613-43f3-a046-4a3e8dcd56db-istiod-ca-cert\") pod \"27643442-2613-43f3-a046-4a3e8dcd56db\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " Oct 31 13:46:41.788468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.788324 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-credential-socket\") pod \"d7327fd4-10e0-41ae-b540-03778f4df456\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " Oct 31 13:46:41.788468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.788358 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-credential-socket\") pod \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " Oct 31 13:46:41.788468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.788391 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7327fd4-10e0-41ae-b540-03778f4df456-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7327fd4-10e0-41ae-b540-03778f4df456-istiod-ca-cert\") pod \"d7327fd4-10e0-41ae-b540-03778f4df456\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " Oct 31 13:46:41.788468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.788422 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27643442-2613-43f3-a046-4a3e8dcd56db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27643442-2613-43f3-a046-4a3e8dcd56db-istio-podinfo\") pod \"27643442-2613-43f3-a046-4a3e8dcd56db\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " Oct 31 13:46:41.788468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.788458 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-certs\") pod \"d01fddda-bc47-40df-914b-d27a8363654e\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " Oct 31 13:46:41.788887 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.788544 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-data\") pod \"27643442-2613-43f3-a046-4a3e8dcd56db\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " Oct 31 13:46:41.788887 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.788634 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-credential-socket\") pod \"d01fddda-bc47-40df-914b-d27a8363654e\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " Oct 31 13:46:41.791056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791029 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-certs\") pod \"27643442-2613-43f3-a046-4a3e8dcd56db\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " Oct 31 13:46:41.791153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791119 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e147e116-ed36-4be3-82ee-b37c85f31665-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e147e116-ed36-4be3-82ee-b37c85f31665-istio-podinfo\") pod \"e147e116-ed36-4be3-82ee-b37c85f31665\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " Oct 31 13:46:41.791206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791160 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-istio-token\" (UniqueName: \"kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-istio-token\") pod \"e147e116-ed36-4be3-82ee-b37c85f31665\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " Oct 31 13:46:41.791256 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791212 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-envoy\") pod \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " Oct 31 13:46:41.791256 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791238 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-data\") pod \"bfe5d1c7-179a-42e0-8457-3cba517be508\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " Oct 31 13:46:41.791360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791316 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-envoy\") pod \"e147e116-ed36-4be3-82ee-b37c85f31665\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " Oct 31 13:46:41.791360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791351 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-certs\") pod \"e147e116-ed36-4be3-82ee-b37c85f31665\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " Oct 31 13:46:41.791469 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791404 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-certs\") pod \"d7327fd4-10e0-41ae-b540-03778f4df456\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " Oct 31 13:46:41.791469 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791438 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-certs\") pod \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " Oct 31 13:46:41.791627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791488 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-credential-socket\") pod \"e147e116-ed36-4be3-82ee-b37c85f31665\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " Oct 31 13:46:41.791627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791529 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d01fddda-bc47-40df-914b-d27a8363654e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d01fddda-bc47-40df-914b-d27a8363654e-istiod-ca-cert\") pod \"d01fddda-bc47-40df-914b-d27a8363654e\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " Oct 31 13:46:41.791627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791559 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-data\") pod \"d01fddda-bc47-40df-914b-d27a8363654e\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " Oct 31 13:46:41.791627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791605 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-kube-api-access-mrkbd\" (UniqueName: \"kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-kube-api-access-mrkbd\") pod \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " Oct 31 13:46:41.791874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791671 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-socket\") pod \"d7327fd4-10e0-41ae-b540-03778f4df456\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " Oct 31 13:46:41.791874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791728 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-kube-api-access-wx5bv\" (UniqueName: \"kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-kube-api-access-wx5bv\") pod \"d7327fd4-10e0-41ae-b540-03778f4df456\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " Oct 31 13:46:41.791874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791756 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-istio-token\" (UniqueName: \"kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-istio-token\") pod \"d01fddda-bc47-40df-914b-d27a8363654e\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " Oct 31 13:46:41.791874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791793 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-podinfo\") pod \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " Oct 31 13:46:41.791874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791831 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istiod-ca-cert\") pod \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " Oct 31 13:46:41.791874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791863 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-token\" (UniqueName: \"kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-token\") pod \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " Oct 31 13:46:41.792103 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791900 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-istio-token\" (UniqueName: \"kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-istio-token\") pod \"27643442-2613-43f3-a046-4a3e8dcd56db\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " Oct 31 13:46:41.792103 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791928 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe5d1c7-179a-42e0-8457-3cba517be508-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe5d1c7-179a-42e0-8457-3cba517be508-istiod-ca-cert\") pod \"bfe5d1c7-179a-42e0-8457-3cba517be508\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " Oct 31 13:46:41.792103 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791957 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e147e116-ed36-4be3-82ee-b37c85f31665-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e147e116-ed36-4be3-82ee-b37c85f31665-istiod-ca-cert\") pod \"e147e116-ed36-4be3-82ee-b37c85f31665\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " Oct 31 13:46:41.792103 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791992 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d01fddda-bc47-40df-914b-d27a8363654e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d01fddda-bc47-40df-914b-d27a8363654e-istio-podinfo\") pod \"d01fddda-bc47-40df-914b-d27a8363654e\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " Oct 31 13:46:41.792103 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792021 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-socket\") pod \"bfe5d1c7-179a-42e0-8457-3cba517be508\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " Oct 31 13:46:41.792103 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792079 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-podinfo\") pod \"bfe5d1c7-179a-42e0-8457-3cba517be508\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " Oct 31 13:46:41.792290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792124 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-data\") pod \"d7327fd4-10e0-41ae-b540-03778f4df456\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " Oct 31 13:46:41.792290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792162 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-credential-socket\") pod \"27643442-2613-43f3-a046-4a3e8dcd56db\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " Oct 31 13:46:41.792290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792226 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-data\") pod \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " Oct 31 13:46:41.792290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792259 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-envoy\") pod \"bfe5d1c7-179a-42e0-8457-3cba517be508\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " Oct 31 13:46:41.792414 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792289 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-data\") pod \"e147e116-ed36-4be3-82ee-b37c85f31665\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " Oct 31 13:46:41.792414 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792371 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-envoy\") pod \"d01fddda-bc47-40df-914b-d27a8363654e\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " Oct 31 13:46:41.792479 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792411 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-kube-api-access-fkm4g\" (UniqueName: \"kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-kube-api-access-fkm4g\") pod \"27643442-2613-43f3-a046-4a3e8dcd56db\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " Oct 31 13:46:41.792479 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792453 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-socket\") pod \"d01fddda-bc47-40df-914b-d27a8363654e\" (UID: \"d01fddda-bc47-40df-914b-d27a8363654e\") " Oct 31 13:46:41.792543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792524 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-envoy\") pod \"d7327fd4-10e0-41ae-b540-03778f4df456\" (UID: \"d7327fd4-10e0-41ae-b540-03778f4df456\") " Oct 31 13:46:41.792649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792625 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-kube-api-access-rbm8h\" (UniqueName: \"kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-kube-api-access-rbm8h\") pod \"e147e116-ed36-4be3-82ee-b37c85f31665\" (UID: \"e147e116-ed36-4be3-82ee-b37c85f31665\") " Oct 31 13:46:41.792783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.787627 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-certs" pod "bfe5d1c7-179a-42e0-8457-3cba517be508" (UID: "bfe5d1c7-179a-42e0-8457-3cba517be508"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.792783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791401 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-kube-api-access-x5k22" pod "bfe5d1c7-179a-42e0-8457-3cba517be508" (UID: "bfe5d1c7-179a-42e0-8457-3cba517be508"). InnerVolumeSpecName "kube-api-access-x5k22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:41.792783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.791662 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-socket" pod "e147e116-ed36-4be3-82ee-b37c85f31665" (UID: "e147e116-ed36-4be3-82ee-b37c85f31665"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.792783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792688 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-kube-api-access-5srcn" pod "d01fddda-bc47-40df-914b-d27a8363654e" (UID: "d01fddda-bc47-40df-914b-d27a8363654e"). InnerVolumeSpecName "kube-api-access-5srcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:41.792783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792736 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-credential-socket\") pod \"bfe5d1c7-179a-42e0-8457-3cba517be508\" (UID: \"bfe5d1c7-179a-42e0-8457-3cba517be508\") " Oct 31 13:46:41.793084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792797 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-socket\") pod \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\" (UID: \"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663\") " Oct 31 13:46:41.793084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792842 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-envoy\") pod \"27643442-2613-43f3-a046-4a3e8dcd56db\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " Oct 31 13:46:41.793084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.792887 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-socket\") pod \"27643442-2613-43f3-a046-4a3e8dcd56db\" (UID: \"27643442-2613-43f3-a046-4a3e8dcd56db\") " Oct 31 13:46:41.798169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.793733 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-istio-token" pod "d7327fd4-10e0-41ae-b540-03778f4df456" (UID: "d7327fd4-10e0-41ae-b540-03778f4df456"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:41.798169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.793912 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.798169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.793934 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5srcn\" (UniqueName: \"kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-kube-api-access-5srcn\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.798169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.793944 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.798169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.793953 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.798169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.793963 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5k22\" (UniqueName: \"kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-kube-api-access-x5k22\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.798169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.794024 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-certs" pod "27643442-2613-43f3-a046-4a3e8dcd56db" (UID: "27643442-2613-43f3-a046-4a3e8dcd56db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.798169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.794158 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-socket" pod "d7327fd4-10e0-41ae-b540-03778f4df456" (UID: "d7327fd4-10e0-41ae-b540-03778f4df456"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.798169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.794838 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7327fd4-10e0-41ae-b540-03778f4df456-istiod-ca-cert" pod "d7327fd4-10e0-41ae-b540-03778f4df456" (UID: "d7327fd4-10e0-41ae-b540-03778f4df456"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:41.798169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.795171 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-socket" pod "d01fddda-bc47-40df-914b-d27a8363654e" (UID: "d01fddda-bc47-40df-914b-d27a8363654e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.798169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.795372 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-certs" pod "e147e116-ed36-4be3-82ee-b37c85f31665" (UID: "e147e116-ed36-4be3-82ee-b37c85f31665"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.798169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.795674 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-data" pod "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" (UID: "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.798169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.795928 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-credential-socket" pod "27643442-2613-43f3-a046-4a3e8dcd56db" (UID: "27643442-2613-43f3-a046-4a3e8dcd56db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.802008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.801916 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-certs" pod "d7327fd4-10e0-41ae-b540-03778f4df456" (UID: "d7327fd4-10e0-41ae-b540-03778f4df456"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.802154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.802135 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-certs" pod "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" (UID: "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.802248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.802155 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27643442-2613-43f3-a046-4a3e8dcd56db-istiod-ca-cert" pod "27643442-2613-43f3-a046-4a3e8dcd56db" (UID: "27643442-2613-43f3-a046-4a3e8dcd56db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:41.802361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.802342 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-credential-socket" pod "e147e116-ed36-4be3-82ee-b37c85f31665" (UID: "e147e116-ed36-4be3-82ee-b37c85f31665"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.802548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.802504 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-credential-socket" pod "d7327fd4-10e0-41ae-b540-03778f4df456" (UID: "d7327fd4-10e0-41ae-b540-03778f4df456"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.802683 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.802665 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d01fddda-bc47-40df-914b-d27a8363654e-istiod-ca-cert" pod "d01fddda-bc47-40df-914b-d27a8363654e" (UID: "d01fddda-bc47-40df-914b-d27a8363654e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:41.802778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.802760 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-credential-socket" pod "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" (UID: "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.802838 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.802824 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7327fd4-10e0-41ae-b540-03778f4df456-istio-podinfo" pod "d7327fd4-10e0-41ae-b540-03778f4df456" (UID: "d7327fd4-10e0-41ae-b540-03778f4df456"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:41.802942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.802926 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-data" pod "d01fddda-bc47-40df-914b-d27a8363654e" (UID: "d01fddda-bc47-40df-914b-d27a8363654e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.803031 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.803016 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-kube-api-access-mrkbd" pod "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" (UID: "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663"). InnerVolumeSpecName "kube-api-access-mrkbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:41.806685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.803473 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-token" pod "bfe5d1c7-179a-42e0-8457-3cba517be508" (UID: "bfe5d1c7-179a-42e0-8457-3cba517be508"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:41.806685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.804442 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-istio-token" pod "d01fddda-bc47-40df-914b-d27a8363654e" (UID: "d01fddda-bc47-40df-914b-d27a8363654e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:41.806685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.805092 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-data" pod "27643442-2613-43f3-a046-4a3e8dcd56db" (UID: "27643442-2613-43f3-a046-4a3e8dcd56db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.806685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.805274 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-certs" pod "d01fddda-bc47-40df-914b-d27a8363654e" (UID: "d01fddda-bc47-40df-914b-d27a8363654e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.806685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.805300 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27643442-2613-43f3-a046-4a3e8dcd56db-istio-podinfo" pod "27643442-2613-43f3-a046-4a3e8dcd56db" (UID: "27643442-2613-43f3-a046-4a3e8dcd56db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:41.806685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.805462 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-credential-socket" pod "d01fddda-bc47-40df-914b-d27a8363654e" (UID: "d01fddda-bc47-40df-914b-d27a8363654e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.806685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.805507 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-data" pod "bfe5d1c7-179a-42e0-8457-3cba517be508" (UID: "bfe5d1c7-179a-42e0-8457-3cba517be508"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.806685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.805768 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e147e116-ed36-4be3-82ee-b37c85f31665-istiod-ca-cert" pod "e147e116-ed36-4be3-82ee-b37c85f31665" (UID: "e147e116-ed36-4be3-82ee-b37c85f31665"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:41.806685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.806143 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istiod-ca-cert" pod "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" (UID: "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:41.810559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.810535 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-envoy" pod "d7327fd4-10e0-41ae-b540-03778f4df456" (UID: "d7327fd4-10e0-41ae-b540-03778f4df456"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.810836 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.810802 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-envoy" pod "e147e116-ed36-4be3-82ee-b37c85f31665" (UID: "e147e116-ed36-4be3-82ee-b37c85f31665"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.813196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.812163 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-envoy" pod "d01fddda-bc47-40df-914b-d27a8363654e" (UID: "d01fddda-bc47-40df-914b-d27a8363654e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.813196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.812405 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-data" pod "e147e116-ed36-4be3-82ee-b37c85f31665" (UID: "e147e116-ed36-4be3-82ee-b37c85f31665"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.813196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.812571 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-socket" pod "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" (UID: "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.813196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.812797 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-credential-socket" pod "bfe5d1c7-179a-42e0-8457-3cba517be508" (UID: "bfe5d1c7-179a-42e0-8457-3cba517be508"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.813196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.812959 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-socket" pod "bfe5d1c7-179a-42e0-8457-3cba517be508" (UID: "bfe5d1c7-179a-42e0-8457-3cba517be508"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.818761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.813507 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-kube-api-access-wx5bv" pod "d7327fd4-10e0-41ae-b540-03778f4df456" (UID: "d7327fd4-10e0-41ae-b540-03778f4df456"). InnerVolumeSpecName "kube-api-access-wx5bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:41.818761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.813568 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e147e116-ed36-4be3-82ee-b37c85f31665-istio-podinfo" pod "e147e116-ed36-4be3-82ee-b37c85f31665" (UID: "e147e116-ed36-4be3-82ee-b37c85f31665"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:41.818761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.813678 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-istio-token" pod "e147e116-ed36-4be3-82ee-b37c85f31665" (UID: "e147e116-ed36-4be3-82ee-b37c85f31665"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:41.818761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.813705 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-kube-api-access-rbm8h" pod "e147e116-ed36-4be3-82ee-b37c85f31665" (UID: "e147e116-ed36-4be3-82ee-b37c85f31665"). InnerVolumeSpecName "kube-api-access-rbm8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:41.818761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.814436 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-socket" pod "27643442-2613-43f3-a046-4a3e8dcd56db" (UID: "27643442-2613-43f3-a046-4a3e8dcd56db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.818761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.814900 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-podinfo" pod "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" (UID: "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:41.818761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.815158 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-envoy" pod "bfe5d1c7-179a-42e0-8457-3cba517be508" (UID: "bfe5d1c7-179a-42e0-8457-3cba517be508"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.818761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.815364 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-data" pod "d7327fd4-10e0-41ae-b540-03778f4df456" (UID: "d7327fd4-10e0-41ae-b540-03778f4df456"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.818761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.815644 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe5d1c7-179a-42e0-8457-3cba517be508-istiod-ca-cert" pod "bfe5d1c7-179a-42e0-8457-3cba517be508" (UID: "bfe5d1c7-179a-42e0-8457-3cba517be508"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:41.818761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.818705 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-podinfo" pod "bfe5d1c7-179a-42e0-8457-3cba517be508" (UID: "bfe5d1c7-179a-42e0-8457-3cba517be508"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:41.818761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.818739 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-istio-token" pod "27643442-2613-43f3-a046-4a3e8dcd56db" (UID: "27643442-2613-43f3-a046-4a3e8dcd56db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:41.819808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.819713 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d01fddda-bc47-40df-914b-d27a8363654e-istio-podinfo" pod "d01fddda-bc47-40df-914b-d27a8363654e" (UID: "d01fddda-bc47-40df-914b-d27a8363654e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:46:41.819808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.819715 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-token" pod "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" (UID: "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:41.819808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.819777 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-kube-api-access-fkm4g" pod "27643442-2613-43f3-a046-4a3e8dcd56db" (UID: "27643442-2613-43f3-a046-4a3e8dcd56db"). InnerVolumeSpecName "kube-api-access-fkm4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:41.819808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.819785 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-envoy" pod "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" (UID: "b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.821044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.820987 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-envoy" pod "27643442-2613-43f3-a046-4a3e8dcd56db" (UID: "27643442-2613-43f3-a046-4a3e8dcd56db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:46:41.895504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895468 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895498 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895509 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7327fd4-10e0-41ae-b540-03778f4df456-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895518 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27643442-2613-43f3-a046-4a3e8dcd56db-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895527 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895538 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895547 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7327fd4-10e0-41ae-b540-03778f4df456-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895555 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27643442-2613-43f3-a046-4a3e8dcd56db-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895563 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895571 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895580 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895588 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895609 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e147e116-ed36-4be3-82ee-b37c85f31665-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895617 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895626 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895635 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895644 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895653 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895662 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895671 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895680 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895689 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d01fddda-bc47-40df-914b-d27a8363654e-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895698 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895706 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrkbd\" (UniqueName: \"kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-kube-api-access-mrkbd\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895716 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895724 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx5bv\" (UniqueName: \"kubernetes.io/projected/d7327fd4-10e0-41ae-b540-03778f4df456-kube-api-access-wx5bv\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895732 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d01fddda-bc47-40df-914b-d27a8363654e-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895740 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895750 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895759 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895766 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895775 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfe5d1c7-179a-42e0-8457-3cba517be508-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895784 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e147e116-ed36-4be3-82ee-b37c85f31665-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895792 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d01fddda-bc47-40df-914b-d27a8363654e-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895800 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895808 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895816 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895824 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895831 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895839 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895847 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e147e116-ed36-4be3-82ee-b37c85f31665-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895855 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895863 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkm4g\" (UniqueName: \"kubernetes.io/projected/27643442-2613-43f3-a046-4a3e8dcd56db-kube-api-access-fkm4g\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.895849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895873 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d01fddda-bc47-40df-914b-d27a8363654e-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.897930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895882 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7327fd4-10e0-41ae-b540-03778f4df456-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.897930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895890 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbm8h\" (UniqueName: \"kubernetes.io/projected/e147e116-ed36-4be3-82ee-b37c85f31665-kube-api-access-rbm8h\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.897930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895898 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfe5d1c7-179a-42e0-8457-3cba517be508-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.897930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895907 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:41.897930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:41.895915 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27643442-2613-43f3-a046-4a3e8dcd56db-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:42.468979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.468901 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" Oct 31 13:46:42.468979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.468902 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4" event={"ID":"27643442-2613-43f3-a046-4a3e8dcd56db","Type":"ContainerDied","Data":"e212db545401692d8a67799d882de69d617dbed27c956677bb50257d5b34fd74"} Oct 31 13:46:42.469452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.468995 5532 scope.go:122] "RemoveContainer" containerID="fb1ef483cd12f6601133838a5ae08a190b326d6a978a9a4edd08a7f7c1185cf9" Oct 31 13:46:42.470652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.470630 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" event={"ID":"b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663","Type":"ContainerDied","Data":"40d92dcdd2414f06c3c9740a234ab7256b2110c7a528999b3e0a29192cd55dce"} Oct 31 13:46:42.470781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.470664 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff" Oct 31 13:46:42.472406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.472389 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" Oct 31 13:46:42.472519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.472389 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh" event={"ID":"d01fddda-bc47-40df-914b-d27a8363654e","Type":"ContainerDied","Data":"7db4ad7efb9ad56b75b64d646f98b29fb0be561bc0b02694d98a9e97790c0c64"} Oct 31 13:46:42.474162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.474140 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" event={"ID":"d7327fd4-10e0-41ae-b540-03778f4df456","Type":"ContainerDied","Data":"400365d717315a12b3e7ba7a66d5591840a39e22c29eb4e3d2a9bd4347e30964"} Oct 31 13:46:42.474268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.474225 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r" Oct 31 13:46:42.475810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.475767 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" Oct 31 13:46:42.475888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.475872 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx" event={"ID":"bfe5d1c7-179a-42e0-8457-3cba517be508","Type":"ContainerDied","Data":"dbef5d3d5368f153d772a42e68f6ad32b15eb99e2ed76f0acf560caed692ee2c"} Oct 31 13:46:42.477635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.477615 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" Oct 31 13:46:42.477753 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.477728 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl" event={"ID":"e147e116-ed36-4be3-82ee-b37c85f31665","Type":"ContainerDied","Data":"d4af76cf47f667235ee1272f2309605e6ff1afa553aef228d7e56387c19bae47"} Oct 31 13:46:42.483321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.483295 5532 scope.go:122] "RemoveContainer" containerID="447a0378fd40567b30c4e43c7bd10ec38c9f0743b947f30919164e60e01fd5c7" Oct 31 13:46:42.495759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.495740 5532 scope.go:122] "RemoveContainer" containerID="e33bae3b70cfc93db66d404d8d84380adf900a5f4004acc52bc89f908a0a99b5" Oct 31 13:46:42.506506 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.506489 5532 scope.go:122] "RemoveContainer" containerID="1f4b5034ac08ec62ebde3d24c3813f8bc650dfdec2e70c0309144c1090285a19" Oct 31 13:46:42.507417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.507393 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4"] Oct 31 13:46:42.512480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.512457 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-t9mj4"] Oct 31 13:46:42.517769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.517750 5532 scope.go:122] "RemoveContainer" containerID="5ee26139b61dc69334a980b4c383da3c7af64013ccb75bbe61c57406599e927a" Oct 31 13:46:42.527983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.527935 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff"] Oct 31 13:46:42.528881 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.528863 5532 scope.go:122] "RemoveContainer" containerID="33cf7bb242187c5c0d325c65385fb019605feb924f1271e15e090eca3c6b17a8" Oct 31 13:46:42.532996 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.532975 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-zmsff"] Oct 31 13:46:42.561006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.560983 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh"] Oct 31 13:46:42.567910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.567890 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-fjvjh"] Oct 31 13:46:42.581022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.581000 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl"] Oct 31 13:46:42.588087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.588062 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-sjmnl"] Oct 31 13:46:42.628616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.622864 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx"] Oct 31 13:46:42.633305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.633269 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-s9zlx"] Oct 31 13:46:42.639773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.639736 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r"] Oct 31 13:46:42.643766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:42.643740 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-61da9d-openshift-default-58d4d4845c-j249r"] Oct 31 13:46:44.006452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:44.006418 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27643442-2613-43f3-a046-4a3e8dcd56db" path="/var/lib/kubelet/pods/27643442-2613-43f3-a046-4a3e8dcd56db/volumes" Oct 31 13:46:44.006937 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:44.006922 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" path="/var/lib/kubelet/pods/b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663/volumes" Oct 31 13:46:44.008018 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:44.007723 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" path="/var/lib/kubelet/pods/bfe5d1c7-179a-42e0-8457-3cba517be508/volumes" Oct 31 13:46:44.008794 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:44.008778 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d01fddda-bc47-40df-914b-d27a8363654e" path="/var/lib/kubelet/pods/d01fddda-bc47-40df-914b-d27a8363654e/volumes" Oct 31 13:46:44.009221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:44.009207 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7327fd4-10e0-41ae-b540-03778f4df456" path="/var/lib/kubelet/pods/d7327fd4-10e0-41ae-b540-03778f4df456/volumes" Oct 31 13:46:44.009740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:44.009716 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e147e116-ed36-4be3-82ee-b37c85f31665" path="/var/lib/kubelet/pods/e147e116-ed36-4be3-82ee-b37c85f31665/volumes" Oct 31 13:46:44.468540 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:44.468502 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg"] Oct 31 13:46:44.488044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:44.488011 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw"] Oct 31 13:46:45.487663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:45.487623 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" podUID="f61f37d8-8b15-4955-bf9d-33a85b280110" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:45.488135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:45.487688 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" podUID="ad4ab732-0660-42a4-80e8-ebed759dd941" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:46.729982 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:46.729950 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r"] Oct 31 13:46:46.743348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:46.743317 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl"] Oct 31 13:46:49.469156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:49.469049 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" podUID="f462da06-0178-4e54-b9bd-ad83be10daf3" containerName="payload-processing" containerID="cri-o://d87006ae1a01ce00767bee32dbfc8b1baef3314e3f85b72edbd9f4a823a37e9b" gracePeriod=25 Oct 31 13:46:49.488724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:49.488643 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" podUID="79003251-6684-4494-9076-fb98c4ac43de" containerName="payload-pre-processing" containerID="cri-o://18bdf0a0282a04f0cc9e45d974cbcfc1ce46738955840da92a01d7ca96cab01c" gracePeriod=25 Oct 31 13:46:49.775903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:49.775876 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" Oct 31 13:46:49.953233 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:49.953194 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f462da06-0178-4e54-b9bd-ad83be10daf3-kube-api-access-2gjxv\" (UniqueName: \"kubernetes.io/projected/f462da06-0178-4e54-b9bd-ad83be10daf3-kube-api-access-2gjxv\") pod \"f462da06-0178-4e54-b9bd-ad83be10daf3\" (UID: \"f462da06-0178-4e54-b9bd-ad83be10daf3\") " Oct 31 13:46:49.953439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:49.953406 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f462da06-0178-4e54-b9bd-ad83be10daf3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f462da06-0178-4e54-b9bd-ad83be10daf3-plugins-config-volume\") pod \"f462da06-0178-4e54-b9bd-ad83be10daf3\" (UID: \"f462da06-0178-4e54-b9bd-ad83be10daf3\") " Oct 31 13:46:49.953801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:49.953777 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f462da06-0178-4e54-b9bd-ad83be10daf3-plugins-config-volume" pod "f462da06-0178-4e54-b9bd-ad83be10daf3" (UID: "f462da06-0178-4e54-b9bd-ad83be10daf3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:49.954017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:49.954001 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f462da06-0178-4e54-b9bd-ad83be10daf3-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:49.955360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:49.955338 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f462da06-0178-4e54-b9bd-ad83be10daf3-kube-api-access-2gjxv" pod "f462da06-0178-4e54-b9bd-ad83be10daf3" (UID: "f462da06-0178-4e54-b9bd-ad83be10daf3"). InnerVolumeSpecName "kube-api-access-2gjxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:50.055856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:50.055823 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gjxv\" (UniqueName: \"kubernetes.io/projected/f462da06-0178-4e54-b9bd-ad83be10daf3-kube-api-access-2gjxv\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:50.563806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:50.563772 5532 generic.go:356] "Generic (PLEG): container finished" podID="f462da06-0178-4e54-b9bd-ad83be10daf3" containerID="d87006ae1a01ce00767bee32dbfc8b1baef3314e3f85b72edbd9f4a823a37e9b" exitCode=0 Oct 31 13:46:50.564254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:50.563835 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" Oct 31 13:46:50.564254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:50.563865 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" event={"ID":"f462da06-0178-4e54-b9bd-ad83be10daf3","Type":"ContainerDied","Data":"d87006ae1a01ce00767bee32dbfc8b1baef3314e3f85b72edbd9f4a823a37e9b"} Oct 31 13:46:50.564254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:50.563925 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg" event={"ID":"f462da06-0178-4e54-b9bd-ad83be10daf3","Type":"ContainerDied","Data":"691019ec66ff5204af56242cc5417103860b13f93d6135704c87b7633a42eee6"} Oct 31 13:46:50.564254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:50.563951 5532 scope.go:122] "RemoveContainer" containerID="d87006ae1a01ce00767bee32dbfc8b1baef3314e3f85b72edbd9f4a823a37e9b" Oct 31 13:46:50.575202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:50.575185 5532 scope.go:122] "RemoveContainer" containerID="d87006ae1a01ce00767bee32dbfc8b1baef3314e3f85b72edbd9f4a823a37e9b" Oct 31 13:46:50.575491 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:46:50.575469 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87006ae1a01ce00767bee32dbfc8b1baef3314e3f85b72edbd9f4a823a37e9b\": container with ID starting with d87006ae1a01ce00767bee32dbfc8b1baef3314e3f85b72edbd9f4a823a37e9b not found: ID does not exist" containerID="d87006ae1a01ce00767bee32dbfc8b1baef3314e3f85b72edbd9f4a823a37e9b" Oct 31 13:46:50.575549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:50.575502 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87006ae1a01ce00767bee32dbfc8b1baef3314e3f85b72edbd9f4a823a37e9b"} err="failed to get container status \"d87006ae1a01ce00767bee32dbfc8b1baef3314e3f85b72edbd9f4a823a37e9b\": rpc error: code = NotFound desc = could not find container \"d87006ae1a01ce00767bee32dbfc8b1baef3314e3f85b72edbd9f4a823a37e9b\": container with ID starting with d87006ae1a01ce00767bee32dbfc8b1baef3314e3f85b72edbd9f4a823a37e9b not found: ID does not exist" Oct 31 13:46:50.604226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:50.604187 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg"] Oct 31 13:46:50.613738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:50.613700 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-cae9ec-7667b7b695-b2lfg"] Oct 31 13:46:51.097384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.097353 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" Oct 31 13:46:51.133093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.133062 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" Oct 31 13:46:51.272899 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.272863 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4ab732-0660-42a4-80e8-ebed759dd941-kube-api-access-lvt4x\" (UniqueName: \"kubernetes.io/projected/ad4ab732-0660-42a4-80e8-ebed759dd941-kube-api-access-lvt4x\") pod \"ad4ab732-0660-42a4-80e8-ebed759dd941\" (UID: \"ad4ab732-0660-42a4-80e8-ebed759dd941\") " Oct 31 13:46:51.273073 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.272958 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f61f37d8-8b15-4955-bf9d-33a85b280110-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61f37d8-8b15-4955-bf9d-33a85b280110-plugins-config-volume\") pod \"f61f37d8-8b15-4955-bf9d-33a85b280110\" (UID: \"f61f37d8-8b15-4955-bf9d-33a85b280110\") " Oct 31 13:46:51.273115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.273095 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4ab732-0660-42a4-80e8-ebed759dd941-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4ab732-0660-42a4-80e8-ebed759dd941-plugins-config-volume\") pod \"ad4ab732-0660-42a4-80e8-ebed759dd941\" (UID: \"ad4ab732-0660-42a4-80e8-ebed759dd941\") " Oct 31 13:46:51.273197 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.273179 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f61f37d8-8b15-4955-bf9d-33a85b280110-kube-api-access-98klb\" (UniqueName: \"kubernetes.io/projected/f61f37d8-8b15-4955-bf9d-33a85b280110-kube-api-access-98klb\") pod \"f61f37d8-8b15-4955-bf9d-33a85b280110\" (UID: \"f61f37d8-8b15-4955-bf9d-33a85b280110\") " Oct 31 13:46:51.273414 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.273385 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f61f37d8-8b15-4955-bf9d-33a85b280110-plugins-config-volume" pod "f61f37d8-8b15-4955-bf9d-33a85b280110" (UID: "f61f37d8-8b15-4955-bf9d-33a85b280110"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:51.273568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.273531 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4ab732-0660-42a4-80e8-ebed759dd941-plugins-config-volume" pod "ad4ab732-0660-42a4-80e8-ebed759dd941" (UID: "ad4ab732-0660-42a4-80e8-ebed759dd941"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:51.273822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.273802 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f61f37d8-8b15-4955-bf9d-33a85b280110-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:51.273880 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.273828 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4ab732-0660-42a4-80e8-ebed759dd941-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:51.275244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.275224 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4ab732-0660-42a4-80e8-ebed759dd941-kube-api-access-lvt4x" pod "ad4ab732-0660-42a4-80e8-ebed759dd941" (UID: "ad4ab732-0660-42a4-80e8-ebed759dd941"). InnerVolumeSpecName "kube-api-access-lvt4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:51.275310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.275252 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f61f37d8-8b15-4955-bf9d-33a85b280110-kube-api-access-98klb" pod "f61f37d8-8b15-4955-bf9d-33a85b280110" (UID: "f61f37d8-8b15-4955-bf9d-33a85b280110"). InnerVolumeSpecName "kube-api-access-98klb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:51.375071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.374994 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvt4x\" (UniqueName: \"kubernetes.io/projected/ad4ab732-0660-42a4-80e8-ebed759dd941-kube-api-access-lvt4x\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:51.375071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.375023 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98klb\" (UniqueName: \"kubernetes.io/projected/f61f37d8-8b15-4955-bf9d-33a85b280110-kube-api-access-98klb\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:51.574573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.574537 5532 generic.go:356] "Generic (PLEG): container finished" podID="ad4ab732-0660-42a4-80e8-ebed759dd941" containerID="dfdc707f45b16b8d3e5df23baae013723448881a433a8e0242248cdc79613eef" exitCode=0 Oct 31 13:46:51.575050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.574586 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" Oct 31 13:46:51.575050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.574626 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" event={"ID":"ad4ab732-0660-42a4-80e8-ebed759dd941","Type":"ContainerDied","Data":"dfdc707f45b16b8d3e5df23baae013723448881a433a8e0242248cdc79613eef"} Oct 31 13:46:51.575050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.574677 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb" event={"ID":"ad4ab732-0660-42a4-80e8-ebed759dd941","Type":"ContainerDied","Data":"725bfc1f7ec756a773d7bf33eaa9e9a7321740c3a829768e7a07de951e1ac325"} Oct 31 13:46:51.575050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.574698 5532 scope.go:122] "RemoveContainer" containerID="dfdc707f45b16b8d3e5df23baae013723448881a433a8e0242248cdc79613eef" Oct 31 13:46:51.581960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.581926 5532 generic.go:356] "Generic (PLEG): container finished" podID="f61f37d8-8b15-4955-bf9d-33a85b280110" containerID="77def19e6fb0460e6d6f6b79ede5dd3230e36d81693c041bde51d99e77f99b54" exitCode=0 Oct 31 13:46:51.582100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.581983 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" Oct 31 13:46:51.582100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.582021 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" event={"ID":"f61f37d8-8b15-4955-bf9d-33a85b280110","Type":"ContainerDied","Data":"77def19e6fb0460e6d6f6b79ede5dd3230e36d81693c041bde51d99e77f99b54"} Oct 31 13:46:51.582100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.582096 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf" event={"ID":"f61f37d8-8b15-4955-bf9d-33a85b280110","Type":"ContainerDied","Data":"740ace4e4b5b4982b612772cc12397763cb4411f61d8b21c78b02eda2d46a0e4"} Oct 31 13:46:51.593343 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.593321 5532 scope.go:122] "RemoveContainer" containerID="dfdc707f45b16b8d3e5df23baae013723448881a433a8e0242248cdc79613eef" Oct 31 13:46:51.593659 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:46:51.593633 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfdc707f45b16b8d3e5df23baae013723448881a433a8e0242248cdc79613eef\": container with ID starting with dfdc707f45b16b8d3e5df23baae013723448881a433a8e0242248cdc79613eef not found: ID does not exist" containerID="dfdc707f45b16b8d3e5df23baae013723448881a433a8e0242248cdc79613eef" Oct 31 13:46:51.593746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.593669 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfdc707f45b16b8d3e5df23baae013723448881a433a8e0242248cdc79613eef"} err="failed to get container status \"dfdc707f45b16b8d3e5df23baae013723448881a433a8e0242248cdc79613eef\": rpc error: code = NotFound desc = could not find container \"dfdc707f45b16b8d3e5df23baae013723448881a433a8e0242248cdc79613eef\": container with ID starting with dfdc707f45b16b8d3e5df23baae013723448881a433a8e0242248cdc79613eef not found: ID does not exist" Oct 31 13:46:51.593746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.593691 5532 scope.go:122] "RemoveContainer" containerID="77def19e6fb0460e6d6f6b79ede5dd3230e36d81693c041bde51d99e77f99b54" Oct 31 13:46:51.603963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.603945 5532 scope.go:122] "RemoveContainer" containerID="77def19e6fb0460e6d6f6b79ede5dd3230e36d81693c041bde51d99e77f99b54" Oct 31 13:46:51.604323 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:46:51.604295 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77def19e6fb0460e6d6f6b79ede5dd3230e36d81693c041bde51d99e77f99b54\": container with ID starting with 77def19e6fb0460e6d6f6b79ede5dd3230e36d81693c041bde51d99e77f99b54 not found: ID does not exist" containerID="77def19e6fb0460e6d6f6b79ede5dd3230e36d81693c041bde51d99e77f99b54" Oct 31 13:46:51.604412 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.604324 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77def19e6fb0460e6d6f6b79ede5dd3230e36d81693c041bde51d99e77f99b54"} err="failed to get container status \"77def19e6fb0460e6d6f6b79ede5dd3230e36d81693c041bde51d99e77f99b54\": rpc error: code = NotFound desc = could not find container \"77def19e6fb0460e6d6f6b79ede5dd3230e36d81693c041bde51d99e77f99b54\": container with ID starting with 77def19e6fb0460e6d6f6b79ede5dd3230e36d81693c041bde51d99e77f99b54 not found: ID does not exist" Oct 31 13:46:51.607435 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.607410 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb"] Oct 31 13:46:51.614391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.614369 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-4fde10-6b7986f6c-9tvcb"] Oct 31 13:46:51.626023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.625972 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf"] Oct 31 13:46:51.631439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.631418 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-4fde10-56488df7f-htwcf"] Oct 31 13:46:51.731086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.731046 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" podUID="7613039d-a5ca-4b20-9c7e-22e4f6fc83ab" containerName="payload-processing" containerID="cri-o://a3a04168097d550b404fc8ce97fdd9b6aca56839ad87e4bb89aacd24a6faabc1" gracePeriod=25 Oct 31 13:46:51.744500 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:51.744467 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" podUID="f0921726-c885-475d-9b36-0c899e6459e1" containerName="payload-pre-processing" containerID="cri-o://1b919f5f2c4a5960fc093f9affeca8f68894413aa51b026e57bb5c80ac40c553" gracePeriod=25 Oct 31 13:46:52.007090 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.007058 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4ab732-0660-42a4-80e8-ebed759dd941" path="/var/lib/kubelet/pods/ad4ab732-0660-42a4-80e8-ebed759dd941/volumes" Oct 31 13:46:52.007567 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.007548 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f462da06-0178-4e54-b9bd-ad83be10daf3" path="/var/lib/kubelet/pods/f462da06-0178-4e54-b9bd-ad83be10daf3/volumes" Oct 31 13:46:52.008037 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.008016 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f61f37d8-8b15-4955-bf9d-33a85b280110" path="/var/lib/kubelet/pods/f61f37d8-8b15-4955-bf9d-33a85b280110/volumes" Oct 31 13:46:52.012755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.012734 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" Oct 31 13:46:52.189626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.189513 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-plugins-config-volume\") pod \"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab\" (UID: \"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab\") " Oct 31 13:46:52.189767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.189727 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-kube-api-access-x7tk9\" (UniqueName: \"kubernetes.io/projected/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-kube-api-access-x7tk9\") pod \"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab\" (UID: \"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab\") " Oct 31 13:46:52.189889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.189867 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-plugins-config-volume" pod "7613039d-a5ca-4b20-9c7e-22e4f6fc83ab" (UID: "7613039d-a5ca-4b20-9c7e-22e4f6fc83ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:46:52.190210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.190191 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:52.191855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.191826 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-kube-api-access-x7tk9" pod "7613039d-a5ca-4b20-9c7e-22e4f6fc83ab" (UID: "7613039d-a5ca-4b20-9c7e-22e4f6fc83ab"). InnerVolumeSpecName "kube-api-access-x7tk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:46:52.291469 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.291435 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7tk9\" (UniqueName: \"kubernetes.io/projected/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab-kube-api-access-x7tk9\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:46:52.593910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.593830 5532 generic.go:356] "Generic (PLEG): container finished" podID="7613039d-a5ca-4b20-9c7e-22e4f6fc83ab" containerID="a3a04168097d550b404fc8ce97fdd9b6aca56839ad87e4bb89aacd24a6faabc1" exitCode=0 Oct 31 13:46:52.593910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.593875 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" Oct 31 13:46:52.594340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.593918 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" event={"ID":"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab","Type":"ContainerDied","Data":"a3a04168097d550b404fc8ce97fdd9b6aca56839ad87e4bb89aacd24a6faabc1"} Oct 31 13:46:52.594340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.593972 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r" event={"ID":"7613039d-a5ca-4b20-9c7e-22e4f6fc83ab","Type":"ContainerDied","Data":"749ddfc2e4512420467bc2f77a21aa3cd08c9c290cce3d651670fcb6192df685"} Oct 31 13:46:52.594340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.593997 5532 scope.go:122] "RemoveContainer" containerID="a3a04168097d550b404fc8ce97fdd9b6aca56839ad87e4bb89aacd24a6faabc1" Oct 31 13:46:52.605572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.605552 5532 scope.go:122] "RemoveContainer" containerID="a3a04168097d550b404fc8ce97fdd9b6aca56839ad87e4bb89aacd24a6faabc1" Oct 31 13:46:52.605914 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:46:52.605896 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3a04168097d550b404fc8ce97fdd9b6aca56839ad87e4bb89aacd24a6faabc1\": container with ID starting with a3a04168097d550b404fc8ce97fdd9b6aca56839ad87e4bb89aacd24a6faabc1 not found: ID does not exist" containerID="a3a04168097d550b404fc8ce97fdd9b6aca56839ad87e4bb89aacd24a6faabc1" Oct 31 13:46:52.605965 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.605922 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3a04168097d550b404fc8ce97fdd9b6aca56839ad87e4bb89aacd24a6faabc1"} err="failed to get container status \"a3a04168097d550b404fc8ce97fdd9b6aca56839ad87e4bb89aacd24a6faabc1\": rpc error: code = NotFound desc = could not find container \"a3a04168097d550b404fc8ce97fdd9b6aca56839ad87e4bb89aacd24a6faabc1\": container with ID starting with a3a04168097d550b404fc8ce97fdd9b6aca56839ad87e4bb89aacd24a6faabc1 not found: ID does not exist" Oct 31 13:46:52.624497 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.624466 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r"] Oct 31 13:46:52.629782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:52.629750 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d835b-5d8fd5d47f-9lk7r"] Oct 31 13:46:53.552572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.552537 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj"] Oct 31 13:46:53.553335 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553316 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f462da06-0178-4e54-b9bd-ad83be10daf3" containerName="payload-processing" Oct 31 13:46:53.553387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553337 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f462da06-0178-4e54-b9bd-ad83be10daf3" containerName="payload-processing" Oct 31 13:46:53.553387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553353 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e147e116-ed36-4be3-82ee-b37c85f31665" containerName="istio-proxy" Oct 31 13:46:53.553387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553358 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e147e116-ed36-4be3-82ee-b37c85f31665" containerName="istio-proxy" Oct 31 13:46:53.553387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553367 5532 cpu_manager.go:414] "containerMap: removing container" podUID="7613039d-a5ca-4b20-9c7e-22e4f6fc83ab" containerName="payload-processing" Oct 31 13:46:53.553387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553372 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7613039d-a5ca-4b20-9c7e-22e4f6fc83ab" containerName="payload-processing" Oct 31 13:46:53.553387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553384 5532 cpu_manager.go:414] "containerMap: removing container" podUID="8a26255b-8277-4bbd-95e9-20afdc2ec6ea" containerName="revoke-keys" Oct 31 13:46:53.553387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553389 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a26255b-8277-4bbd-95e9-20afdc2ec6ea" containerName="revoke-keys" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553402 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f61f37d8-8b15-4955-bf9d-33a85b280110" containerName="payload-pre-processing" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553407 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f61f37d8-8b15-4955-bf9d-33a85b280110" containerName="payload-pre-processing" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553419 5532 cpu_manager.go:414] "containerMap: removing container" podUID="27643442-2613-43f3-a046-4a3e8dcd56db" containerName="istio-proxy" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553424 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27643442-2613-43f3-a046-4a3e8dcd56db" containerName="istio-proxy" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553433 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d01fddda-bc47-40df-914b-d27a8363654e" containerName="istio-proxy" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553437 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d01fddda-bc47-40df-914b-d27a8363654e" containerName="istio-proxy" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553444 5532 cpu_manager.go:414] "containerMap: removing container" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerName="istio-proxy" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553448 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerName="istio-proxy" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553459 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" containerName="istio-proxy" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553464 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" containerName="istio-proxy" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553471 5532 cpu_manager.go:414] "containerMap: removing container" podUID="7cb2933a-c77e-42d9-b3ba-adad4aa5bb14" containerName="revoke-keys" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553476 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7cb2933a-c77e-42d9-b3ba-adad4aa5bb14" containerName="revoke-keys" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553487 5532 cpu_manager.go:414] "containerMap: removing container" podUID="3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" containerName="istio-proxy" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553492 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" containerName="istio-proxy" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553512 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" containerName="istio-proxy" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553527 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" containerName="istio-proxy" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553542 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ad4ab732-0660-42a4-80e8-ebed759dd941" containerName="payload-processing" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553548 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4ab732-0660-42a4-80e8-ebed759dd941" containerName="payload-processing" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553562 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d7327fd4-10e0-41ae-b540-03778f4df456" containerName="istio-proxy" Oct 31 13:46:53.553623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553567 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7327fd4-10e0-41ae-b540-03778f4df456" containerName="istio-proxy" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553705 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a26255b-8277-4bbd-95e9-20afdc2ec6ea" containerName="revoke-keys" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553719 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f61f37d8-8b15-4955-bf9d-33a85b280110" containerName="payload-pre-processing" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553725 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="27643442-2613-43f3-a046-4a3e8dcd56db" containerName="istio-proxy" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553733 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d01fddda-bc47-40df-914b-d27a8363654e" containerName="istio-proxy" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553741 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe5d1c7-179a-42e0-8457-3cba517be508" containerName="istio-proxy" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553748 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8a86ac3-9fe8-4d9b-b5c9-8669fc03e663" containerName="istio-proxy" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553754 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="7cb2933a-c77e-42d9-b3ba-adad4aa5bb14" containerName="revoke-keys" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553761 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dde3d07-0294-4137-a7f2-4fd2e4a5feb4" containerName="istio-proxy" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553773 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e95899d5-7d0f-4e7e-a964-f6b2b71ae9d6" containerName="istio-proxy" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553787 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4ab732-0660-42a4-80e8-ebed759dd941" containerName="payload-processing" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553794 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7327fd4-10e0-41ae-b540-03778f4df456" containerName="istio-proxy" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553800 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f462da06-0178-4e54-b9bd-ad83be10daf3" containerName="payload-processing" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553806 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e147e116-ed36-4be3-82ee-b37c85f31665" containerName="istio-proxy" Oct 31 13:46:53.554215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.553813 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="7613039d-a5ca-4b20-9c7e-22e4f6fc83ab" containerName="payload-processing" Oct 31 13:46:53.559455 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.559435 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" Oct 31 13:46:53.570877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.570847 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-97927b-dockercfg-csxbp\"" Oct 31 13:46:53.571589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.571097 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-97927b\"" Oct 31 13:46:53.577531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.577497 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj"] Oct 31 13:46:53.590905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.590870 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m"] Oct 31 13:46:53.595970 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.595951 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" Oct 31 13:46:53.601940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.601912 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m"] Oct 31 13:46:53.730620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.714810 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c48eb37-c893-4cbf-b24a-9db54e12d699-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m\" (UID: \"5c48eb37-c893-4cbf-b24a-9db54e12d699\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" Oct 31 13:46:53.730620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.715070 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v5kw\" (UniqueName: \"kubernetes.io/projected/f0267300-2667-4d7a-8335-fafbc3a4b889-kube-api-access-5v5kw\") pod \"payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj\" (UID: \"f0267300-2667-4d7a-8335-fafbc3a4b889\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" Oct 31 13:46:53.730620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.715141 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0267300-2667-4d7a-8335-fafbc3a4b889-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj\" (UID: \"f0267300-2667-4d7a-8335-fafbc3a4b889\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" Oct 31 13:46:53.730620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.715185 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn7jj\" (UniqueName: \"kubernetes.io/projected/5c48eb37-c893-4cbf-b24a-9db54e12d699-kube-api-access-mn7jj\") pod \"payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m\" (UID: \"5c48eb37-c893-4cbf-b24a-9db54e12d699\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" Oct 31 13:46:53.817670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.816698 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v5kw\" (UniqueName: \"kubernetes.io/projected/f0267300-2667-4d7a-8335-fafbc3a4b889-kube-api-access-5v5kw\") pod \"payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj\" (UID: \"f0267300-2667-4d7a-8335-fafbc3a4b889\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" Oct 31 13:46:53.817670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.816766 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0267300-2667-4d7a-8335-fafbc3a4b889-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj\" (UID: \"f0267300-2667-4d7a-8335-fafbc3a4b889\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" Oct 31 13:46:53.817670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.816798 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn7jj\" (UniqueName: \"kubernetes.io/projected/5c48eb37-c893-4cbf-b24a-9db54e12d699-kube-api-access-mn7jj\") pod \"payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m\" (UID: \"5c48eb37-c893-4cbf-b24a-9db54e12d699\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" Oct 31 13:46:53.817670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.816856 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c48eb37-c893-4cbf-b24a-9db54e12d699-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m\" (UID: \"5c48eb37-c893-4cbf-b24a-9db54e12d699\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" Oct 31 13:46:53.817670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.817583 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c48eb37-c893-4cbf-b24a-9db54e12d699-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m\" (UID: \"5c48eb37-c893-4cbf-b24a-9db54e12d699\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" Oct 31 13:46:53.819567 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.818531 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0267300-2667-4d7a-8335-fafbc3a4b889-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj\" (UID: \"f0267300-2667-4d7a-8335-fafbc3a4b889\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" Oct 31 13:46:53.833558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.833519 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn7jj\" (UniqueName: \"kubernetes.io/projected/5c48eb37-c893-4cbf-b24a-9db54e12d699-kube-api-access-mn7jj\") pod \"payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m\" (UID: \"5c48eb37-c893-4cbf-b24a-9db54e12d699\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" Oct 31 13:46:53.833723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.833519 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v5kw\" (UniqueName: \"kubernetes.io/projected/f0267300-2667-4d7a-8335-fafbc3a4b889-kube-api-access-5v5kw\") pod \"payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj\" (UID: \"f0267300-2667-4d7a-8335-fafbc3a4b889\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" Oct 31 13:46:53.891279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.891242 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" Oct 31 13:46:53.940614 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:53.940569 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" Oct 31 13:46:54.010689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:54.010556 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7613039d-a5ca-4b20-9c7e-22e4f6fc83ab" path="/var/lib/kubelet/pods/7613039d-a5ca-4b20-9c7e-22e4f6fc83ab/volumes" Oct 31 13:46:54.181189 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:54.181158 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj"] Oct 31 13:46:54.181579 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:46:54.181549 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0267300_2667_4d7a_8335_fafbc3a4b889.slice/crio-95145a33ed2a2846ae924f446213d6da8e58314addd812ce9ee764c912912406 WatchSource:0}: Error finding container 95145a33ed2a2846ae924f446213d6da8e58314addd812ce9ee764c912912406: Status 404 returned error can't find the container with id 95145a33ed2a2846ae924f446213d6da8e58314addd812ce9ee764c912912406 Oct 31 13:46:54.253733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:54.253705 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m"] Oct 31 13:46:54.256042 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:46:54.256007 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c48eb37_c893_4cbf_b24a_9db54e12d699.slice/crio-20b54a531e283dc64fe1b607a639e0f1695a3a6b040f47e7e5847293e9285036 WatchSource:0}: Error finding container 20b54a531e283dc64fe1b607a639e0f1695a3a6b040f47e7e5847293e9285036: Status 404 returned error can't find the container with id 20b54a531e283dc64fe1b607a639e0f1695a3a6b040f47e7e5847293e9285036 Oct 31 13:46:54.619548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:54.619451 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" event={"ID":"5c48eb37-c893-4cbf-b24a-9db54e12d699","Type":"ContainerStarted","Data":"f4813f6d5d1910f59369c751044cd17b804f976637e301adb06dd869f1617478"} Oct 31 13:46:54.619548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:54.619518 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" event={"ID":"5c48eb37-c893-4cbf-b24a-9db54e12d699","Type":"ContainerStarted","Data":"20b54a531e283dc64fe1b607a639e0f1695a3a6b040f47e7e5847293e9285036"} Oct 31 13:46:54.619548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:54.619541 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" Oct 31 13:46:54.621341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:54.621316 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" event={"ID":"f0267300-2667-4d7a-8335-fafbc3a4b889","Type":"ContainerStarted","Data":"b0c782305a3441affae92f162767035b51cdb8c6de89f8fb3be83414f3886889"} Oct 31 13:46:54.621463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:54.621361 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" event={"ID":"f0267300-2667-4d7a-8335-fafbc3a4b889","Type":"ContainerStarted","Data":"95145a33ed2a2846ae924f446213d6da8e58314addd812ce9ee764c912912406"} Oct 31 13:46:54.621463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:54.621388 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" Oct 31 13:46:54.641259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:54.641209 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" podStartSLOduration=1.6411947489999998 podStartE2EDuration="1.641194749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:46:54.64095967 +0000 UTC m=+2582.786703498" watchObservedRunningTime="2026-10-31 13:46:54.641194749 +0000 UTC m=+2582.786938595" Oct 31 13:46:54.664163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:54.664104 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" podStartSLOduration=1.664087624 podStartE2EDuration="1.664087624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:46:54.660583291 +0000 UTC m=+2582.806327119" watchObservedRunningTime="2026-10-31 13:46:54.664087624 +0000 UTC m=+2582.809831456" Oct 31 13:46:54.976588 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:54.976548 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" podUID="f0921726-c885-475d-9b36-0c899e6459e1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:46:55.381992 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:55.381891 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67"] Oct 31 13:46:55.397151 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:55.397114 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cae9ec-bcp67"] Oct 31 13:46:56.006030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:56.005997 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a26255b-8277-4bbd-95e9-20afdc2ec6ea" path="/var/lib/kubelet/pods/8a26255b-8277-4bbd-95e9-20afdc2ec6ea/volumes" Oct 31 13:46:58.102220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:46:58.102182 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" podUID="79003251-6684-4494-9076-fb98c4ac43de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:47:00.605936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:00.604262 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6"] Oct 31 13:47:00.622356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:00.622323 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d835b-zrls6"] Oct 31 13:47:01.840257 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:47:01.840205 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:47:02.040105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:02.040055 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7cb2933a-c77e-42d9-b3ba-adad4aa5bb14" path="/var/lib/kubelet/pods/7cb2933a-c77e-42d9-b3ba-adad4aa5bb14/volumes" Oct 31 13:47:04.977560 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:04.977520 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" podUID="f0921726-c885-475d-9b36-0c899e6459e1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:47:05.043120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:05.043083 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf"] Oct 31 13:47:05.043418 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:05.043355 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" podUID="e5ba4078-c9a3-4f92-933e-d6ef817c11aa" containerName="istio-proxy" containerID="cri-o://147f136c4404154dbc6098038e91e712e05334656abf72cda91d3747e503149b" gracePeriod=30 Oct 31 13:47:05.633466 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:05.633436 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" Oct 31 13:47:05.633637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:05.633512 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" Oct 31 13:47:08.102550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:08.102505 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" podUID="79003251-6684-4494-9076-fb98c4ac43de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:47:09.771234 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.771204 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" Oct 31 13:47:09.837952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.833144 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79003251-6684-4494-9076-fb98c4ac43de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79003251-6684-4494-9076-fb98c4ac43de-plugins-config-volume\") pod \"79003251-6684-4494-9076-fb98c4ac43de\" (UID: \"79003251-6684-4494-9076-fb98c4ac43de\") " Oct 31 13:47:09.837952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.833251 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79003251-6684-4494-9076-fb98c4ac43de-kube-api-access-t89pq\" (UniqueName: \"kubernetes.io/projected/79003251-6684-4494-9076-fb98c4ac43de-kube-api-access-t89pq\") pod \"79003251-6684-4494-9076-fb98c4ac43de\" (UID: \"79003251-6684-4494-9076-fb98c4ac43de\") " Oct 31 13:47:09.837952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.834295 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79003251-6684-4494-9076-fb98c4ac43de-plugins-config-volume" pod "79003251-6684-4494-9076-fb98c4ac43de" (UID: "79003251-6684-4494-9076-fb98c4ac43de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:47:09.837952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.837156 5532 generic.go:356] "Generic (PLEG): container finished" podID="79003251-6684-4494-9076-fb98c4ac43de" containerID="18bdf0a0282a04f0cc9e45d974cbcfc1ce46738955840da92a01d7ca96cab01c" exitCode=0 Oct 31 13:47:09.837952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.837282 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" event={"ID":"79003251-6684-4494-9076-fb98c4ac43de","Type":"ContainerDied","Data":"18bdf0a0282a04f0cc9e45d974cbcfc1ce46738955840da92a01d7ca96cab01c"} Oct 31 13:47:09.837952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.837337 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" event={"ID":"79003251-6684-4494-9076-fb98c4ac43de","Type":"ContainerDied","Data":"a8a0242b2c35dc1b66300971586d641bf7dd6908920bb716e9aac17edc289dc3"} Oct 31 13:47:09.837952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.837368 5532 scope.go:122] "RemoveContainer" containerID="18bdf0a0282a04f0cc9e45d974cbcfc1ce46738955840da92a01d7ca96cab01c" Oct 31 13:47:09.837952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.837541 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw" Oct 31 13:47:09.859585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.859540 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79003251-6684-4494-9076-fb98c4ac43de-kube-api-access-t89pq" pod "79003251-6684-4494-9076-fb98c4ac43de" (UID: "79003251-6684-4494-9076-fb98c4ac43de"). InnerVolumeSpecName "kube-api-access-t89pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:09.863263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.863236 5532 scope.go:122] "RemoveContainer" containerID="18bdf0a0282a04f0cc9e45d974cbcfc1ce46738955840da92a01d7ca96cab01c" Oct 31 13:47:09.863984 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:47:09.863951 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18bdf0a0282a04f0cc9e45d974cbcfc1ce46738955840da92a01d7ca96cab01c\": container with ID starting with 18bdf0a0282a04f0cc9e45d974cbcfc1ce46738955840da92a01d7ca96cab01c not found: ID does not exist" containerID="18bdf0a0282a04f0cc9e45d974cbcfc1ce46738955840da92a01d7ca96cab01c" Oct 31 13:47:09.864127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.863995 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18bdf0a0282a04f0cc9e45d974cbcfc1ce46738955840da92a01d7ca96cab01c"} err="failed to get container status \"18bdf0a0282a04f0cc9e45d974cbcfc1ce46738955840da92a01d7ca96cab01c\": rpc error: code = NotFound desc = could not find container \"18bdf0a0282a04f0cc9e45d974cbcfc1ce46738955840da92a01d7ca96cab01c\": container with ID starting with 18bdf0a0282a04f0cc9e45d974cbcfc1ce46738955840da92a01d7ca96cab01c not found: ID does not exist" Oct 31 13:47:09.935692 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.935659 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/79003251-6684-4494-9076-fb98c4ac43de-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:09.935692 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:09.935688 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t89pq\" (UniqueName: \"kubernetes.io/projected/79003251-6684-4494-9076-fb98c4ac43de-kube-api-access-t89pq\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:10.161742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.161711 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw"] Oct 31 13:47:10.168515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.168490 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-cae9ec-67f455b699-d7jmw"] Oct 31 13:47:10.330930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.330903 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:47:10.341380 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.341352 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-credential-socket\") pod \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " Oct 31 13:47:10.341521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.341401 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-token\" (UniqueName: \"kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-token\") pod \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " Oct 31 13:47:10.341521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.341456 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-podinfo\") pod \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " Oct 31 13:47:10.341521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.341509 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-data\") pod \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " Oct 31 13:47:10.341687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.341589 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-credential-socket" pod "e5ba4078-c9a3-4f92-933e-d6ef817c11aa" (UID: "e5ba4078-c9a3-4f92-933e-d6ef817c11aa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:10.341687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.341665 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-certs\") pod \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " Oct 31 13:47:10.341755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.341708 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-kube-api-access-m2g6n\" (UniqueName: \"kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-kube-api-access-m2g6n\") pod \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " Oct 31 13:47:10.341755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.341740 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-envoy\") pod \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " Oct 31 13:47:10.341839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.341775 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istiod-ca-cert\") pod \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " Oct 31 13:47:10.341839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.341813 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-data" pod "e5ba4078-c9a3-4f92-933e-d6ef817c11aa" (UID: "e5ba4078-c9a3-4f92-933e-d6ef817c11aa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:10.341944 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.341857 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-socket\") pod \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\" (UID: \"e5ba4078-c9a3-4f92-933e-d6ef817c11aa\") " Oct 31 13:47:10.342167 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.342142 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-socket" pod "e5ba4078-c9a3-4f92-933e-d6ef817c11aa" (UID: "e5ba4078-c9a3-4f92-933e-d6ef817c11aa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:10.342284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.342218 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-certs" pod "e5ba4078-c9a3-4f92-933e-d6ef817c11aa" (UID: "e5ba4078-c9a3-4f92-933e-d6ef817c11aa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:10.342284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.342235 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istiod-ca-cert" pod "e5ba4078-c9a3-4f92-933e-d6ef817c11aa" (UID: "e5ba4078-c9a3-4f92-933e-d6ef817c11aa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:47:10.342624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.342585 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:10.342723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.342629 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:10.342723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.342644 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:10.342723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.342661 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:10.342723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.342674 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:10.344041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.344014 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-token" pod "e5ba4078-c9a3-4f92-933e-d6ef817c11aa" (UID: "e5ba4078-c9a3-4f92-933e-d6ef817c11aa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:10.344130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.344057 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-podinfo" pod "e5ba4078-c9a3-4f92-933e-d6ef817c11aa" (UID: "e5ba4078-c9a3-4f92-933e-d6ef817c11aa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:47:10.344130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.344100 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-kube-api-access-m2g6n" pod "e5ba4078-c9a3-4f92-933e-d6ef817c11aa" (UID: "e5ba4078-c9a3-4f92-933e-d6ef817c11aa"). InnerVolumeSpecName "kube-api-access-m2g6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:10.344760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.344742 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-envoy" pod "e5ba4078-c9a3-4f92-933e-d6ef817c11aa" (UID: "e5ba4078-c9a3-4f92-933e-d6ef817c11aa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:47:10.444358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.444322 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m2g6n\" (UniqueName: \"kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-kube-api-access-m2g6n\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:10.444358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.444355 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:10.444572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.444368 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:10.444572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.444381 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5ba4078-c9a3-4f92-933e-d6ef817c11aa-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:10.524642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.524588 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2"] Oct 31 13:47:10.526218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.525859 5532 cpu_manager.go:414] "containerMap: removing container" podUID="79003251-6684-4494-9076-fb98c4ac43de" containerName="payload-pre-processing" Oct 31 13:47:10.526218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.525881 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79003251-6684-4494-9076-fb98c4ac43de" containerName="payload-pre-processing" Oct 31 13:47:10.526218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.525950 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e5ba4078-c9a3-4f92-933e-d6ef817c11aa" containerName="istio-proxy" Oct 31 13:47:10.526218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.525958 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ba4078-c9a3-4f92-933e-d6ef817c11aa" containerName="istio-proxy" Oct 31 13:47:10.526218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.526105 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ba4078-c9a3-4f92-933e-d6ef817c11aa" containerName="istio-proxy" Oct 31 13:47:10.526218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.526126 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="79003251-6684-4494-9076-fb98c4ac43de" containerName="payload-pre-processing" Oct 31 13:47:10.531089 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.531055 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.542776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.541793 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2"] Oct 31 13:47:10.543424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.543320 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-974711-openshift-default-dockercfg-86mwq\"" Oct 31 13:47:10.548056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.545470 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-certs\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.548056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.545522 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjvw7\" (UniqueName: \"kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-kube-api-access-cjvw7\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.548056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.545579 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-envoy\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.548056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.545751 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f63ce148-84ae-4429-9661-77b0e688205a-istiod-ca-cert\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.548056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.545829 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-credential-socket\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.548056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.545881 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-istio-token\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.548056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.545942 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-socket\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.548056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.546004 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f63ce148-84ae-4429-9661-77b0e688205a-istio-podinfo\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.548056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.546071 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-data\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.650098 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-data\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.650211 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-certs\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.650248 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjvw7\" (UniqueName: \"kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-kube-api-access-cjvw7\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.650314 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-envoy\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.650429 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f63ce148-84ae-4429-9661-77b0e688205a-istiod-ca-cert\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.650506 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-credential-socket\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.650555 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-istio-token\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.650630 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-socket\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.650697 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f63ce148-84ae-4429-9661-77b0e688205a-istio-podinfo\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.669683 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-certs\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.669936 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-data\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.670166 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-credential-socket\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.671281 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f63ce148-84ae-4429-9661-77b0e688205a-istiod-ca-cert\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.673636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.671975 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-socket\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.703619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.702763 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-envoy\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.723690 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.721206 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f63ce148-84ae-4429-9661-77b0e688205a-istio-podinfo\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.741223 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.728930 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjvw7\" (UniqueName: \"kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-kube-api-access-cjvw7\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.742016 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.741988 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-istio-token\") pod \"e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.850963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.850930 5532 generic.go:356] "Generic (PLEG): container finished" podID="e5ba4078-c9a3-4f92-933e-d6ef817c11aa" containerID="147f136c4404154dbc6098038e91e712e05334656abf72cda91d3747e503149b" exitCode=0 Oct 31 13:47:10.851427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.850988 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" Oct 31 13:47:10.851427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.851021 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" event={"ID":"e5ba4078-c9a3-4f92-933e-d6ef817c11aa","Type":"ContainerDied","Data":"147f136c4404154dbc6098038e91e712e05334656abf72cda91d3747e503149b"} Oct 31 13:47:10.851427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.851079 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf" event={"ID":"e5ba4078-c9a3-4f92-933e-d6ef817c11aa","Type":"ContainerDied","Data":"116931040d1f70539646b6012a34e04ea1ea3702408e057de6b8fa953469a528"} Oct 31 13:47:10.851427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.851109 5532 scope.go:122] "RemoveContainer" containerID="147f136c4404154dbc6098038e91e712e05334656abf72cda91d3747e503149b" Oct 31 13:47:10.856898 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.856877 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:10.863066 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.863051 5532 scope.go:122] "RemoveContainer" containerID="147f136c4404154dbc6098038e91e712e05334656abf72cda91d3747e503149b" Oct 31 13:47:10.863313 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:47:10.863297 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"147f136c4404154dbc6098038e91e712e05334656abf72cda91d3747e503149b\": container with ID starting with 147f136c4404154dbc6098038e91e712e05334656abf72cda91d3747e503149b not found: ID does not exist" containerID="147f136c4404154dbc6098038e91e712e05334656abf72cda91d3747e503149b" Oct 31 13:47:10.863377 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.863323 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"147f136c4404154dbc6098038e91e712e05334656abf72cda91d3747e503149b"} err="failed to get container status \"147f136c4404154dbc6098038e91e712e05334656abf72cda91d3747e503149b\": rpc error: code = NotFound desc = could not find container \"147f136c4404154dbc6098038e91e712e05334656abf72cda91d3747e503149b\": container with ID starting with 147f136c4404154dbc6098038e91e712e05334656abf72cda91d3747e503149b not found: ID does not exist" Oct 31 13:47:10.887101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.887069 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf"] Oct 31 13:47:10.897914 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:10.897888 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-cae9ec-openshift-default-758c77f6d5-j8vdf"] Oct 31 13:47:11.176303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:11.176275 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2"] Oct 31 13:47:11.176816 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:47:11.176793 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf63ce148_84ae_4429_9661_77b0e688205a.slice/crio-95ce09366bfbd703905ae4ebcc8d024bac1b04fb953b730b4b8fe1004ee66658 WatchSource:0}: Error finding container 95ce09366bfbd703905ae4ebcc8d024bac1b04fb953b730b4b8fe1004ee66658: Status 404 returned error can't find the container with id 95ce09366bfbd703905ae4ebcc8d024bac1b04fb953b730b4b8fe1004ee66658 Oct 31 13:47:11.179521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:11.179490 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:11.179622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:11.179563 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:11.179622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:11.179607 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:11.861642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:11.861585 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" event={"ID":"f63ce148-84ae-4429-9661-77b0e688205a","Type":"ContainerStarted","Data":"9b373aaf8cbb57afa0a8f1b18d6cc3f77c8ac5b7ccd326f879b7f86f3afb8a20"} Oct 31 13:47:11.862101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:11.861674 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" event={"ID":"f63ce148-84ae-4429-9661-77b0e688205a","Type":"ContainerStarted","Data":"95ce09366bfbd703905ae4ebcc8d024bac1b04fb953b730b4b8fe1004ee66658"} Oct 31 13:47:11.895221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:11.895156 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" podStartSLOduration=1.89513637 podStartE2EDuration="1.89513637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:11.892550979 +0000 UTC m=+2600.038294819" watchObservedRunningTime="2026-10-31 13:47:11.89513637 +0000 UTC m=+2600.040880203" Oct 31 13:47:11.903947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:11.903916 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk"] Oct 31 13:47:11.908718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:11.908692 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" Oct 31 13:47:11.916668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:11.916098 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-a39e7e\"" Oct 31 13:47:11.919013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:11.918986 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk"] Oct 31 13:47:11.992325 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:11.992284 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4bae26-45a2-4f55-b30f-31fb08d18578-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk\" (UID: \"8e4bae26-45a2-4f55-b30f-31fb08d18578\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" Oct 31 13:47:11.992512 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:11.992366 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqzvc\" (UniqueName: \"kubernetes.io/projected/8e4bae26-45a2-4f55-b30f-31fb08d18578-kube-api-access-bqzvc\") pod \"payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk\" (UID: \"8e4bae26-45a2-4f55-b30f-31fb08d18578\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" Oct 31 13:47:12.006115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.006031 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79003251-6684-4494-9076-fb98c4ac43de" path="/var/lib/kubelet/pods/79003251-6684-4494-9076-fb98c4ac43de/volumes" Oct 31 13:47:12.006615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.006584 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ba4078-c9a3-4f92-933e-d6ef817c11aa" path="/var/lib/kubelet/pods/e5ba4078-c9a3-4f92-933e-d6ef817c11aa/volumes" Oct 31 13:47:12.094346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.094305 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" Oct 31 13:47:12.094729 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.094702 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4bae26-45a2-4f55-b30f-31fb08d18578-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk\" (UID: \"8e4bae26-45a2-4f55-b30f-31fb08d18578\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" Oct 31 13:47:12.094862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.094775 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqzvc\" (UniqueName: \"kubernetes.io/projected/8e4bae26-45a2-4f55-b30f-31fb08d18578-kube-api-access-bqzvc\") pod \"payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk\" (UID: \"8e4bae26-45a2-4f55-b30f-31fb08d18578\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" Oct 31 13:47:12.095548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.095520 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4bae26-45a2-4f55-b30f-31fb08d18578-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk\" (UID: \"8e4bae26-45a2-4f55-b30f-31fb08d18578\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" Oct 31 13:47:12.116639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.115091 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqzvc\" (UniqueName: \"kubernetes.io/projected/8e4bae26-45a2-4f55-b30f-31fb08d18578-kube-api-access-bqzvc\") pod \"payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk\" (UID: \"8e4bae26-45a2-4f55-b30f-31fb08d18578\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" Oct 31 13:47:12.196778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.196744 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0921726-c885-475d-9b36-0c899e6459e1-kube-api-access-txn8m\" (UniqueName: \"kubernetes.io/projected/f0921726-c885-475d-9b36-0c899e6459e1-kube-api-access-txn8m\") pod \"f0921726-c885-475d-9b36-0c899e6459e1\" (UID: \"f0921726-c885-475d-9b36-0c899e6459e1\") " Oct 31 13:47:12.197038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.196805 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0921726-c885-475d-9b36-0c899e6459e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0921726-c885-475d-9b36-0c899e6459e1-plugins-config-volume\") pod \"f0921726-c885-475d-9b36-0c899e6459e1\" (UID: \"f0921726-c885-475d-9b36-0c899e6459e1\") " Oct 31 13:47:12.198357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.198096 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0921726-c885-475d-9b36-0c899e6459e1-plugins-config-volume" pod "f0921726-c885-475d-9b36-0c899e6459e1" (UID: "f0921726-c885-475d-9b36-0c899e6459e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:47:12.202019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.201975 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0921726-c885-475d-9b36-0c899e6459e1-kube-api-access-txn8m" pod "f0921726-c885-475d-9b36-0c899e6459e1" (UID: "f0921726-c885-475d-9b36-0c899e6459e1"). InnerVolumeSpecName "kube-api-access-txn8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:47:12.228124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.228094 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" Oct 31 13:47:12.300804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.300710 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txn8m\" (UniqueName: \"kubernetes.io/projected/f0921726-c885-475d-9b36-0c899e6459e1-kube-api-access-txn8m\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:12.300804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.300735 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0921726-c885-475d-9b36-0c899e6459e1-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:47:12.514711 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:47:12.514677 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4bae26_45a2_4f55_b30f_31fb08d18578.slice/crio-3765cfcd68f703576122400d7f4de665994995693218ed6ca9d36969571be3b3 WatchSource:0}: Error finding container 3765cfcd68f703576122400d7f4de665994995693218ed6ca9d36969571be3b3: Status 404 returned error can't find the container with id 3765cfcd68f703576122400d7f4de665994995693218ed6ca9d36969571be3b3 Oct 31 13:47:12.516212 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.516186 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk"] Oct 31 13:47:12.857350 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.857272 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:12.863608 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.863570 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:12.872121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.872096 5532 generic.go:356] "Generic (PLEG): container finished" podID="f0921726-c885-475d-9b36-0c899e6459e1" containerID="1b919f5f2c4a5960fc093f9affeca8f68894413aa51b026e57bb5c80ac40c553" exitCode=0 Oct 31 13:47:12.872236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.872155 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" Oct 31 13:47:12.872236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.872171 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" event={"ID":"f0921726-c885-475d-9b36-0c899e6459e1","Type":"ContainerDied","Data":"1b919f5f2c4a5960fc093f9affeca8f68894413aa51b026e57bb5c80ac40c553"} Oct 31 13:47:12.872236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.872216 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl" event={"ID":"f0921726-c885-475d-9b36-0c899e6459e1","Type":"ContainerDied","Data":"39d936dd905e9b28a8078ace7033b8e2e1a20ccfde12b606e862b06d1119461d"} Oct 31 13:47:12.872401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.872241 5532 scope.go:122] "RemoveContainer" containerID="1b919f5f2c4a5960fc093f9affeca8f68894413aa51b026e57bb5c80ac40c553" Oct 31 13:47:12.874313 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.874292 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" event={"ID":"8e4bae26-45a2-4f55-b30f-31fb08d18578","Type":"ContainerStarted","Data":"b160164005b866ebc0335b943057fef8043c99ed1d5ba37a20fb3df43176fa55"} Oct 31 13:47:12.874603 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.874579 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" Oct 31 13:47:12.874728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.874639 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:12.874728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.874658 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" event={"ID":"8e4bae26-45a2-4f55-b30f-31fb08d18578","Type":"ContainerStarted","Data":"3765cfcd68f703576122400d7f4de665994995693218ed6ca9d36969571be3b3"} Oct 31 13:47:12.875488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.875467 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:47:12.886862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.886842 5532 scope.go:122] "RemoveContainer" containerID="1b919f5f2c4a5960fc093f9affeca8f68894413aa51b026e57bb5c80ac40c553" Oct 31 13:47:12.887199 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:47:12.887179 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b919f5f2c4a5960fc093f9affeca8f68894413aa51b026e57bb5c80ac40c553\": container with ID starting with 1b919f5f2c4a5960fc093f9affeca8f68894413aa51b026e57bb5c80ac40c553 not found: ID does not exist" containerID="1b919f5f2c4a5960fc093f9affeca8f68894413aa51b026e57bb5c80ac40c553" Oct 31 13:47:12.887257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.887206 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b919f5f2c4a5960fc093f9affeca8f68894413aa51b026e57bb5c80ac40c553"} err="failed to get container status \"1b919f5f2c4a5960fc093f9affeca8f68894413aa51b026e57bb5c80ac40c553\": rpc error: code = NotFound desc = could not find container \"1b919f5f2c4a5960fc093f9affeca8f68894413aa51b026e57bb5c80ac40c553\": container with ID starting with 1b919f5f2c4a5960fc093f9affeca8f68894413aa51b026e57bb5c80ac40c553 not found: ID does not exist" Oct 31 13:47:12.940156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.940118 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl"] Oct 31 13:47:12.954832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.954800 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d835b-545c644787-crtpl"] Oct 31 13:47:12.972669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:12.972615 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" podStartSLOduration=1.972582797 podStartE2EDuration="1.972582797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:12.97031274 +0000 UTC m=+2601.116056563" watchObservedRunningTime="2026-10-31 13:47:12.972582797 +0000 UTC m=+2601.118326625" Oct 31 13:47:14.005404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:14.005367 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0921726-c885-475d-9b36-0c899e6459e1" path="/var/lib/kubelet/pods/f0921726-c885-475d-9b36-0c899e6459e1/volumes" Oct 31 13:47:15.699743 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:15.699711 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk"] Oct 31 13:47:15.700457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:15.700442 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f0921726-c885-475d-9b36-0c899e6459e1" containerName="payload-pre-processing" Oct 31 13:47:15.700510 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:15.700458 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0921726-c885-475d-9b36-0c899e6459e1" containerName="payload-pre-processing" Oct 31 13:47:15.700637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:15.700587 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0921726-c885-475d-9b36-0c899e6459e1" containerName="payload-pre-processing" Oct 31 13:47:15.705663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:15.705640 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" Oct 31 13:47:15.716876 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:15.716840 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-974711\"" Oct 31 13:47:15.719869 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:15.719844 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk"] Oct 31 13:47:15.775030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:15.774992 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzlc6\" (UniqueName: \"kubernetes.io/projected/8ad79fea-9683-4d69-a99b-77e69db4bb68-kube-api-access-jzlc6\") pod \"payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk\" (UID: \"8ad79fea-9683-4d69-a99b-77e69db4bb68\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" Oct 31 13:47:15.775264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:15.775250 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad79fea-9683-4d69-a99b-77e69db4bb68-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk\" (UID: \"8ad79fea-9683-4d69-a99b-77e69db4bb68\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" Oct 31 13:47:15.893967 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:15.891579 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad79fea-9683-4d69-a99b-77e69db4bb68-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk\" (UID: \"8ad79fea-9683-4d69-a99b-77e69db4bb68\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" Oct 31 13:47:15.893967 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:15.891752 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzlc6\" (UniqueName: \"kubernetes.io/projected/8ad79fea-9683-4d69-a99b-77e69db4bb68-kube-api-access-jzlc6\") pod \"payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk\" (UID: \"8ad79fea-9683-4d69-a99b-77e69db4bb68\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" Oct 31 13:47:15.893967 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:15.893225 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad79fea-9683-4d69-a99b-77e69db4bb68-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk\" (UID: \"8ad79fea-9683-4d69-a99b-77e69db4bb68\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" Oct 31 13:47:15.927633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:15.927168 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzlc6\" (UniqueName: \"kubernetes.io/projected/8ad79fea-9683-4d69-a99b-77e69db4bb68-kube-api-access-jzlc6\") pod \"payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk\" (UID: \"8ad79fea-9683-4d69-a99b-77e69db4bb68\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" Oct 31 13:47:16.024476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:16.024383 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" Oct 31 13:47:16.314108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:16.314065 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk"] Oct 31 13:47:16.314267 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:47:16.314216 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad79fea_9683_4d69_a99b_77e69db4bb68.slice/crio-8cd15dd477f51263f4472d5edf67d35b7be78713315a3047ae569ca0df8dc705 WatchSource:0}: Error finding container 8cd15dd477f51263f4472d5edf67d35b7be78713315a3047ae569ca0df8dc705: Status 404 returned error can't find the container with id 8cd15dd477f51263f4472d5edf67d35b7be78713315a3047ae569ca0df8dc705 Oct 31 13:47:16.963152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:16.963112 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" event={"ID":"8ad79fea-9683-4d69-a99b-77e69db4bb68","Type":"ContainerStarted","Data":"21c4eb83f644f0d81a17c405c5d3144269e2945413cf830b2ebeb9460f367cd8"} Oct 31 13:47:16.963555 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:16.963179 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" event={"ID":"8ad79fea-9683-4d69-a99b-77e69db4bb68","Type":"ContainerStarted","Data":"8cd15dd477f51263f4472d5edf67d35b7be78713315a3047ae569ca0df8dc705"} Oct 31 13:47:16.963555 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:16.963209 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" Oct 31 13:47:16.984174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:16.984113 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" podStartSLOduration=1.984100351 podStartE2EDuration="1.984100351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:16.982588539 +0000 UTC m=+2605.128332379" watchObservedRunningTime="2026-10-31 13:47:16.984100351 +0000 UTC m=+2605.129844219" Oct 31 13:47:23.887244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:23.887212 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" Oct 31 13:47:27.974582 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:27.974551 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" Oct 31 13:47:30.701450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.701360 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms"] Oct 31 13:47:30.708623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.708018 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:30.728789 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.728705 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms"] Oct 31 13:47:30.729194 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.729057 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-a39e7e-serving-cert\"" Oct 31 13:47:30.827898 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.827864 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:30.828071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.827936 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2536955f-80a6-4959-bc32-8b56a38f5631-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:30.828264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.828236 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pthsq\" (UniqueName: \"kubernetes.io/projected/2536955f-80a6-4959-bc32-8b56a38f5631-kube-api-access-pthsq\") pod \"maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:30.828364 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.828348 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:30.930662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.929984 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:30.930662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.930085 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:30.930662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.930165 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2536955f-80a6-4959-bc32-8b56a38f5631-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:30.930662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.930421 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pthsq\" (UniqueName: \"kubernetes.io/projected/2536955f-80a6-4959-bc32-8b56a38f5631-kube-api-access-pthsq\") pod \"maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:30.931508 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.931475 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2536955f-80a6-4959-bc32-8b56a38f5631-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:30.933199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.933173 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:30.933355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.933330 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:30.945820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:30.945794 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pthsq\" (UniqueName: \"kubernetes.io/projected/2536955f-80a6-4959-bc32-8b56a38f5631-kube-api-access-pthsq\") pod \"maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:31.050465 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:31.050388 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:31.368667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:31.351334 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms"] Oct 31 13:47:31.423644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:31.420136 5532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:47:31.762975 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:47:31.762938 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:47:32.083228 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:32.083135 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" event={"ID":"2536955f-80a6-4959-bc32-8b56a38f5631","Type":"ContainerStarted","Data":"91505def6b7265d15eb1cc06a334d41350be333c574c641c9284ccb693fb0440"} Oct 31 13:47:32.083228 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:32.083193 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" event={"ID":"2536955f-80a6-4959-bc32-8b56a38f5631","Type":"ContainerStarted","Data":"a30d4cd7f7a78537f22a224848741085a7b1f7e0958c3bfb4a5f05888e903c86"} Oct 31 13:47:32.083464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:32.083360 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:32.107247 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:32.107196 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" podStartSLOduration=1.696859028 podStartE2EDuration="2.107181914s" totalImagesPullingTime="410.322886ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:32.106065986 +0000 UTC m=+2620.251809813" watchObservedRunningTime="2026-10-31 13:47:32.107181914 +0000 UTC m=+2620.252925741" Oct 31 13:47:35.595493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.595445 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87"] Oct 31 13:47:35.603940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.603911 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.611587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.611561 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-612b46\"/\"openshift-service-ca.crt\"" Oct 31 13:47:35.611806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.611679 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-612b46\"/\"kube-root-ca.crt\"" Oct 31 13:47:35.612802 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.612778 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-612b46\"/\"auth-test-model-612b46-kserve-self-signed-certs\"" Oct 31 13:47:35.612950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.612942 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-612b46\"/\"default-dockercfg-wwhsx\"" Oct 31 13:47:35.613137 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.613114 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87"] Oct 31 13:47:35.714241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.714206 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e487890-211d-4588-90c6-52499fc28ce0-tls-certs\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.714419 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.714267 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-model-cache\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.714419 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.714364 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-dshm\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.714499 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.714460 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-home\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.714540 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.714519 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-tmp-dir\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.714717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.714697 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xzld\" (UniqueName: \"kubernetes.io/projected/8e487890-211d-4588-90c6-52499fc28ce0-kube-api-access-2xzld\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.816558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.816387 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xzld\" (UniqueName: \"kubernetes.io/projected/8e487890-211d-4588-90c6-52499fc28ce0-kube-api-access-2xzld\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.816558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.816494 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e487890-211d-4588-90c6-52499fc28ce0-tls-certs\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.816558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.816547 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-model-cache\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.816945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.816625 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-dshm\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.816945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.816737 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-home\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.816945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.816808 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-tmp-dir\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.817644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.817551 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-tmp-dir\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.817870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.817849 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-model-cache\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.818278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.818246 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-home\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.821359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.821318 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e487890-211d-4588-90c6-52499fc28ce0-tls-certs\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.822033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.822008 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-dshm\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.831488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.831456 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xzld\" (UniqueName: \"kubernetes.io/projected/8e487890-211d-4588-90c6-52499fc28ce0-kube-api-access-2xzld\") pod \"auth-test-model-612b46-kserve-6f94769ddf-gvc87\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:35.922900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:35.922865 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:36.218983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:36.218953 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87"] Oct 31 13:47:36.219455 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:47:36.219433 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e487890_211d_4588_90c6_52499fc28ce0.slice/crio-2de2043f3b3e3b18f1fb838c0715798380d5369c36222747c25e6d3d23cee76b WatchSource:0}: Error finding container 2de2043f3b3e3b18f1fb838c0715798380d5369c36222747c25e6d3d23cee76b: Status 404 returned error can't find the container with id 2de2043f3b3e3b18f1fb838c0715798380d5369c36222747c25e6d3d23cee76b Oct 31 13:47:37.130875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:37.130838 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" event={"ID":"8e487890-211d-4588-90c6-52499fc28ce0","Type":"ContainerStarted","Data":"195a9e1017bc68407880d033a43715911179e4d9cf3d7637eae56f8af265dc89"} Oct 31 13:47:37.131345 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:37.130887 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" event={"ID":"8e487890-211d-4588-90c6-52499fc28ce0","Type":"ContainerStarted","Data":"2de2043f3b3e3b18f1fb838c0715798380d5369c36222747c25e6d3d23cee76b"} Oct 31 13:47:37.175619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:37.157462 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" podStartSLOduration=2.157440626 podStartE2EDuration="2.157440626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:37.154554344 +0000 UTC m=+2625.300298176" watchObservedRunningTime="2026-10-31 13:47:37.157440626 +0000 UTC m=+2625.303184454" Oct 31 13:47:37.923115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:37.923081 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:38.096653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:38.096624 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 13:47:43.357448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:43.357412 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh"] Oct 31 13:47:43.363224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:43.363201 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" Oct 31 13:47:43.384770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:43.384736 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh"] Oct 31 13:47:43.401396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:43.396054 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-599291-dockercfg-bzs5h\"" Oct 31 13:47:43.445644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:43.445615 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-599291\"" Oct 31 13:47:43.457495 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:43.457459 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4l8h\" (UniqueName: \"kubernetes.io/projected/b4497eeb-2f9a-4131-b3e4-3fcef824c637-kube-api-access-f4l8h\") pod \"payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh\" (UID: \"b4497eeb-2f9a-4131-b3e4-3fcef824c637\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" Oct 31 13:47:43.457673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:43.457523 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4497eeb-2f9a-4131-b3e4-3fcef824c637-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh\" (UID: \"b4497eeb-2f9a-4131-b3e4-3fcef824c637\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" Oct 31 13:47:43.559479 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:43.559439 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4l8h\" (UniqueName: \"kubernetes.io/projected/b4497eeb-2f9a-4131-b3e4-3fcef824c637-kube-api-access-f4l8h\") pod \"payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh\" (UID: \"b4497eeb-2f9a-4131-b3e4-3fcef824c637\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" Oct 31 13:47:43.559735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:43.559517 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4497eeb-2f9a-4131-b3e4-3fcef824c637-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh\" (UID: \"b4497eeb-2f9a-4131-b3e4-3fcef824c637\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" Oct 31 13:47:43.560791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:43.560761 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4497eeb-2f9a-4131-b3e4-3fcef824c637-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh\" (UID: \"b4497eeb-2f9a-4131-b3e4-3fcef824c637\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" Oct 31 13:47:43.612390 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:43.604880 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4l8h\" (UniqueName: \"kubernetes.io/projected/b4497eeb-2f9a-4131-b3e4-3fcef824c637-kube-api-access-f4l8h\") pod \"payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh\" (UID: \"b4497eeb-2f9a-4131-b3e4-3fcef824c637\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" Oct 31 13:47:43.693749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:43.693711 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" Oct 31 13:47:44.005753 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:47:44.005719 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4497eeb_2f9a_4131_b3e4_3fcef824c637.slice/crio-9397105b4b37c441598417ba6caa688379ffe6bb1c91fe407028bb12ae0cb79f WatchSource:0}: Error finding container 9397105b4b37c441598417ba6caa688379ffe6bb1c91fe407028bb12ae0cb79f: Status 404 returned error can't find the container with id 9397105b4b37c441598417ba6caa688379ffe6bb1c91fe407028bb12ae0cb79f Oct 31 13:47:44.007671 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:44.007643 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh"] Oct 31 13:47:44.217140 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:44.217109 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" event={"ID":"b4497eeb-2f9a-4131-b3e4-3fcef824c637","Type":"ContainerStarted","Data":"09e29943cee80b121099b7438491b95e63ede76290297e477a5cf0d99d5882f0"} Oct 31 13:47:44.217337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:44.217160 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" event={"ID":"b4497eeb-2f9a-4131-b3e4-3fcef824c637","Type":"ContainerStarted","Data":"9397105b4b37c441598417ba6caa688379ffe6bb1c91fe407028bb12ae0cb79f"} Oct 31 13:47:44.217337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:44.217190 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" Oct 31 13:47:44.239095 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:44.239047 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" podStartSLOduration=1.239033416 podStartE2EDuration="1.239033416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:44.237681644 +0000 UTC m=+2632.383425473" watchObservedRunningTime="2026-10-31 13:47:44.239033416 +0000 UTC m=+2632.384777246" Oct 31 13:47:45.923305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:45.923269 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:45.937581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:45.937545 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:46.250936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:46.250849 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:47:53.816293 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:53.816257 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:47:55.234486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:55.234460 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" Oct 31 13:47:57.366077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.366037 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s"] Oct 31 13:47:57.388625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.385796 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s"] Oct 31 13:47:57.388625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.385925 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.429076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.429034 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-credential-socket\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.429265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.429129 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-certs\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.429265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.429157 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-token\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.429385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.429269 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-socket\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.429385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.429302 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-podinfo\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.429492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.429380 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf8n9\" (UniqueName: \"kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-kube-api-access-wf8n9\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.429492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.429424 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-data\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.429625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.429563 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5eaae20-98cf-46cd-a44f-74d780a567cd-istiod-ca-cert\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.429625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.429588 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-envoy\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.489980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.489938 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-35d13e-openshift-default-dockercfg-wwk8w\"" Oct 31 13:47:57.531052 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.531020 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf8n9\" (UniqueName: \"kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-kube-api-access-wf8n9\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.531235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.531077 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-data\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.531235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.531216 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5eaae20-98cf-46cd-a44f-74d780a567cd-istiod-ca-cert\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.531354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.531243 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-envoy\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.531354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.531289 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-credential-socket\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.531550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.531361 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-certs\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.531550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.531379 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-token\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.531550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.531489 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-socket\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.531550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.531520 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-podinfo\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.531550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.531541 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-data\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.533244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.531791 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-credential-socket\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.533244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.531926 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-certs\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.533244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.532149 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-socket\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.533244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.532574 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5eaae20-98cf-46cd-a44f-74d780a567cd-istiod-ca-cert\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.534214 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.534189 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-envoy\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.536012 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.535990 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-podinfo\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.583989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.583899 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf8n9\" (UniqueName: \"kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-kube-api-access-wf8n9\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.584508 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.584471 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-token\") pod \"e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:57.782942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:57.782910 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:58.096476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:58.096444 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s"] Oct 31 13:47:58.097185 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:47:58.097154 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5eaae20_98cf_46cd_a44f_74d780a567cd.slice/crio-ae32d915ba89ef4f29224a94f742df6249d45257eb2eaa1b94995a8e0cb3adc8 WatchSource:0}: Error finding container ae32d915ba89ef4f29224a94f742df6249d45257eb2eaa1b94995a8e0cb3adc8: Status 404 returned error can't find the container with id ae32d915ba89ef4f29224a94f742df6249d45257eb2eaa1b94995a8e0cb3adc8 Oct 31 13:47:58.100211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:58.100180 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:58.100279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:58.100245 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:58.100316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:58.100278 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:47:58.405858 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:58.405136 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" event={"ID":"d5eaae20-98cf-46cd-a44f-74d780a567cd","Type":"ContainerStarted","Data":"26a440d1c1402f46d8873ef3a74b4f9df2ac0fa1b0258ecbad8bd84561da4f3f"} Oct 31 13:47:58.405858 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:58.405198 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" event={"ID":"d5eaae20-98cf-46cd-a44f-74d780a567cd","Type":"ContainerStarted","Data":"ae32d915ba89ef4f29224a94f742df6249d45257eb2eaa1b94995a8e0cb3adc8"} Oct 31 13:47:58.435003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:58.434947 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" podStartSLOduration=1.434933402 podStartE2EDuration="1.434933402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:47:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:47:58.430546532 +0000 UTC m=+2646.576290362" watchObservedRunningTime="2026-10-31 13:47:58.434933402 +0000 UTC m=+2646.580677229" Oct 31 13:47:58.783798 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:58.783716 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:47:59.790017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:47:59.789983 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:48:00.424544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:00.424407 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:48:00.425478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:00.425458 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:48:01.037141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.037106 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh"] Oct 31 13:48:01.068631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.067890 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.068631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.068071 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh"] Oct 31 13:48:01.075427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.075245 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-14ba5a-openshift-default-dockercfg-qj6n9\"" Oct 31 13:48:01.124566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.120005 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv798\" (UniqueName: \"kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-kube-api-access-tv798\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.124566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.120102 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-token\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.124566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.120198 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1242bc05-fa46-4097-afaf-36a1e19194a9-istiod-ca-cert\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.124566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.120257 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-podinfo\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.124566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.120336 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-envoy\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.124566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.120424 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-credential-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.124566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.120493 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-certs\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.124566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.120572 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-data\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.124566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.120691 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.252625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.239722 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1242bc05-fa46-4097-afaf-36a1e19194a9-istiod-ca-cert\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.252625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.239806 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-podinfo\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.252625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.239878 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-envoy\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.252625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.239963 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-credential-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.252625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.240013 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-certs\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.252625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.240084 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-data\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.252625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.240123 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.252625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.240232 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv798\" (UniqueName: \"kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-kube-api-access-tv798\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.252625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.240280 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-token\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.268618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.257426 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1242bc05-fa46-4097-afaf-36a1e19194a9-istiod-ca-cert\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.268618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.258187 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-certs\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.270799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.269341 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-data\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.270799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.269968 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-credential-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.270799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.269983 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.293058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.292951 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-token\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.293796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.293714 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-podinfo\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.309982 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.302993 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-envoy\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.311526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.310865 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv798\" (UniqueName: \"kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-kube-api-access-tv798\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.392980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.392825 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:01.730766 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.729565 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh"] Oct 31 13:48:01.745271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.744757 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:01.745271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.744855 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:01.745271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:01.744900 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:01.778129 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:48:01.778093 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:48:02.444712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:02.444627 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" event={"ID":"1242bc05-fa46-4097-afaf-36a1e19194a9","Type":"ContainerStarted","Data":"229d94709835a6fe0570be3d7d56ce5f087c0f3fee03cbd2879862be3baa12b1"} Oct 31 13:48:02.444712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:02.444680 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" event={"ID":"1242bc05-fa46-4097-afaf-36a1e19194a9","Type":"ContainerStarted","Data":"afdf345218603bc956a245f38a1955892ce1e174fd50985d2235d21e78402999"} Oct 31 13:48:03.393759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:03.393719 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:03.401097 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:03.401067 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:03.428986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:03.428930 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" podStartSLOduration=3.428913235 podStartE2EDuration="3.428913235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:02.473119509 +0000 UTC m=+2650.618863340" watchObservedRunningTime="2026-10-31 13:48:03.428913235 +0000 UTC m=+2651.574657064" Oct 31 13:48:03.454819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:03.454788 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:03.455791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:03.455760 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:48:06.481094 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:06.481058 5532 generic.go:356] "Generic (PLEG): container finished" podID="f63ce148-84ae-4429-9661-77b0e688205a" containerID="9b373aaf8cbb57afa0a8f1b18d6cc3f77c8ac5b7ccd326f879b7f86f3afb8a20" exitCode=0 Oct 31 13:48:06.481524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:06.481135 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" event={"ID":"f63ce148-84ae-4429-9661-77b0e688205a","Type":"ContainerDied","Data":"9b373aaf8cbb57afa0a8f1b18d6cc3f77c8ac5b7ccd326f879b7f86f3afb8a20"} Oct 31 13:48:06.481691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:06.481674 5532 scope.go:122] "RemoveContainer" containerID="9b373aaf8cbb57afa0a8f1b18d6cc3f77c8ac5b7ccd326f879b7f86f3afb8a20" Oct 31 13:48:06.483756 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:06.483720 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:06.483863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:06.483803 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:06.483863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:06.483836 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:07.495293 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:07.495255 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" event={"ID":"f63ce148-84ae-4429-9661-77b0e688205a","Type":"ContainerStarted","Data":"c33cc72858300afe60d86c3981ceba0c38a044e698a8c2d41b308acde7e8a51c"} Oct 31 13:48:07.858003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:07.857914 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:48:07.858279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:07.858237 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 13:48:07.858372 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:07.858342 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 13:48:08.857468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:08.857428 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 13:48:08.866639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:08.858213 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 13:48:09.858716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:09.858678 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 13:48:09.859294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:09.859258 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 13:48:10.859617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:10.859031 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 13:48:10.859617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:10.859095 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 13:48:11.435265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.435233 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g"] Oct 31 13:48:11.461935 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.461898 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.469898 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.469866 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-6a3533-openshift-default-dockercfg-79xkn\"" Oct 31 13:48:11.474035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.474006 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g"] Oct 31 13:48:11.570506 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.570473 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-token\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.570757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.570639 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istiod-ca-cert\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.570757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.570707 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-credential-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.570757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.570737 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-podinfo\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.570949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.570779 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-data\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.570949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.570857 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-envoy\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.570949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.570895 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-certs\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.571108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.570947 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdn5\" (UniqueName: \"kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-kube-api-access-spdn5\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.571170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.571123 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.672796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.672759 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.673013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.672849 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-token\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.673013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.672907 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istiod-ca-cert\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.673013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.672960 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-credential-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.673013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.672993 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-podinfo\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.673176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.673037 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-data\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.673176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.673149 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-envoy\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.673240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.673187 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-certs\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.673282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.673238 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdn5\" (UniqueName: \"kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-kube-api-access-spdn5\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.673474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.673447 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.673582 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.673467 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-credential-socket\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.673582 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.673529 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-data\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.673745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.673636 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-certs\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.674131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.674105 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istiod-ca-cert\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.675803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.675746 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-envoy\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.675991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.675971 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-podinfo\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.688134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.688078 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-token\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.688287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.688268 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdn5\" (UniqueName: \"kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-kube-api-access-spdn5\") pod \"e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.791321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.791283 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:11.860486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.860019 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 13:48:11.860486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:11.860083 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 13:48:12.138177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:12.138146 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g"] Oct 31 13:48:12.169412 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:12.169377 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:12.169529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:12.169444 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:12.169529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:12.169473 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:48:12.564621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:12.550361 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" event={"ID":"80c8a0f2-8f1d-4733-8a30-d4852d045a0d","Type":"ContainerStarted","Data":"7efa7a86a57b783648d2d743db94d4e318ef15715fcd1892097dad47682766ee"} Oct 31 13:48:12.564621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:12.550432 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" event={"ID":"80c8a0f2-8f1d-4733-8a30-d4852d045a0d","Type":"ContainerStarted","Data":"0aa7cd7cb8bf17ab00e12000da21a9af7f4c2fed01ecf4d930b658026cbf175b"} Oct 31 13:48:12.582325 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:12.581283 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" podStartSLOduration=1.5812642609999998 podStartE2EDuration="1.581264261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:12.578011599 +0000 UTC m=+2660.723755429" watchObservedRunningTime="2026-10-31 13:48:12.581264261 +0000 UTC m=+2660.727008189" Oct 31 13:48:12.791986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:12.791953 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:12.859677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:12.858743 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 31 13:48:12.859677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:12.858802 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 31 13:48:12.875298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:12.875266 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:48:13.793827 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:13.793795 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 13:48:13.794011 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:13.793845 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" podUID="80c8a0f2-8f1d-4733-8a30-d4852d045a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 13:48:13.867108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:13.867070 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:48:13.868304 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:13.868280 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 13:48:14.792301 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:14.792266 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 13:48:14.792777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:14.792322 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" podUID="80c8a0f2-8f1d-4733-8a30-d4852d045a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 13:48:15.791699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:15.791661 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 13:48:15.791876 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:15.791719 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" podUID="80c8a0f2-8f1d-4733-8a30-d4852d045a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 13:48:16.792581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:16.792545 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 13:48:16.793006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:16.792609 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" podUID="80c8a0f2-8f1d-4733-8a30-d4852d045a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 13:48:17.792636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:17.792559 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 13:48:17.793085 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:17.792665 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" podUID="80c8a0f2-8f1d-4733-8a30-d4852d045a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 13:48:18.792604 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:18.792568 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 31 13:48:18.792782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:18.792662 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" podUID="80c8a0f2-8f1d-4733-8a30-d4852d045a0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 31 13:48:19.796866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:19.796837 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:19.797308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:19.797294 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:19.798160 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:19.798140 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:48:23.799315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:23.799268 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd"] Oct 31 13:48:23.806740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:23.806710 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:23.812454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:23.812420 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-35d13e-serving-cert\"" Oct 31 13:48:23.816629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:23.816570 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd"] Oct 31 13:48:23.917972 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:23.917934 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4686420e-4a64-47bd-8f9b-c442fe8eda2d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:23.918142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:23.918046 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:23.918226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:23.918204 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:23.918347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:23.918254 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfbqs\" (UniqueName: \"kubernetes.io/projected/4686420e-4a64-47bd-8f9b-c442fe8eda2d-kube-api-access-jfbqs\") pod \"maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:24.024308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:24.024096 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:24.024653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:24.024574 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfbqs\" (UniqueName: \"kubernetes.io/projected/4686420e-4a64-47bd-8f9b-c442fe8eda2d-kube-api-access-jfbqs\") pod \"maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:24.025023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:24.024999 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4686420e-4a64-47bd-8f9b-c442fe8eda2d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:24.025215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:24.025197 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:24.026044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:24.026019 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4686420e-4a64-47bd-8f9b-c442fe8eda2d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:24.042264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:24.037967 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:24.042264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:24.041142 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:24.044467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:24.044408 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfbqs\" (UniqueName: \"kubernetes.io/projected/4686420e-4a64-47bd-8f9b-c442fe8eda2d-kube-api-access-jfbqs\") pod \"maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:24.131263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:24.131186 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:24.447183 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:24.447150 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd"] Oct 31 13:48:24.447639 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:48:24.447559 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4686420e_4a64_47bd_8f9b_c442fe8eda2d.slice/crio-3e7489dd8e35e5c9a9cc0d928cb25b05cade99eb13b90f2f47afa3ed21c37432 WatchSource:0}: Error finding container 3e7489dd8e35e5c9a9cc0d928cb25b05cade99eb13b90f2f47afa3ed21c37432: Status 404 returned error can't find the container with id 3e7489dd8e35e5c9a9cc0d928cb25b05cade99eb13b90f2f47afa3ed21c37432 Oct 31 13:48:24.673829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:24.673794 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" event={"ID":"4686420e-4a64-47bd-8f9b-c442fe8eda2d","Type":"ContainerStarted","Data":"3e7489dd8e35e5c9a9cc0d928cb25b05cade99eb13b90f2f47afa3ed21c37432"} Oct 31 13:48:25.683664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:25.683628 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" event={"ID":"4686420e-4a64-47bd-8f9b-c442fe8eda2d","Type":"ContainerStarted","Data":"71e924efb9b6f9d04540efba29bb80758e4ae04394f8d14c16e1dac748bb1d65"} Oct 31 13:48:25.684064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:25.683773 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:25.707572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:25.707515 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" podStartSLOduration=2.27938705 podStartE2EDuration="2.707498055s" totalImagesPullingTime="428.111005ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:48:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:48:25.706472998 +0000 UTC m=+2673.852216820" watchObservedRunningTime="2026-10-31 13:48:25.707498055 +0000 UTC m=+2673.853241886" Oct 31 13:48:31.697936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:31.697897 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 13:48:31.764541 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:48:31.764502 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:48:52.661574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:48:52.661542 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:49:01.466769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:01.466726 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:49:01.467175 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:01.466811 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:49:01.786621 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:49:01.786485 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:49:04.497795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:04.497702 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:49:04.497795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:04.497776 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:49:05.189799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:05.189756 5532 generic.go:356] "Generic (PLEG): container finished" podID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerID="26a440d1c1402f46d8873ef3a74b4f9df2ac0fa1b0258ecbad8bd84561da4f3f" exitCode=0 Oct 31 13:49:05.189988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:05.189829 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" event={"ID":"d5eaae20-98cf-46cd-a44f-74d780a567cd","Type":"ContainerDied","Data":"26a440d1c1402f46d8873ef3a74b4f9df2ac0fa1b0258ecbad8bd84561da4f3f"} Oct 31 13:49:05.190306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:05.190282 5532 scope.go:122] "RemoveContainer" containerID="26a440d1c1402f46d8873ef3a74b4f9df2ac0fa1b0258ecbad8bd84561da4f3f" Oct 31 13:49:05.191759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:05.191732 5532 generic.go:356] "Generic (PLEG): container finished" podID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerID="229d94709835a6fe0570be3d7d56ce5f087c0f3fee03cbd2879862be3baa12b1" exitCode=0 Oct 31 13:49:05.191889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:05.191802 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" event={"ID":"1242bc05-fa46-4097-afaf-36a1e19194a9","Type":"ContainerDied","Data":"229d94709835a6fe0570be3d7d56ce5f087c0f3fee03cbd2879862be3baa12b1"} Oct 31 13:49:05.192208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:05.192189 5532 scope.go:122] "RemoveContainer" containerID="229d94709835a6fe0570be3d7d56ce5f087c0f3fee03cbd2879862be3baa12b1" Oct 31 13:49:05.192306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:05.192278 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:49:05.192366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:05.192340 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:49:05.192417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:05.192372 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:49:05.193815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:05.193781 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:49:05.193913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:05.193844 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:49:05.193913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:05.193876 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:49:06.202705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:06.202669 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" event={"ID":"d5eaae20-98cf-46cd-a44f-74d780a567cd","Type":"ContainerStarted","Data":"7bbbca30ce49d48ccb4315e9c248a56eee0204e455a4868d99d8d153c627e582"} Oct 31 13:49:06.205070 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:06.205043 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" event={"ID":"1242bc05-fa46-4097-afaf-36a1e19194a9","Type":"ContainerStarted","Data":"f48790afd708ac55b0e686cdfe985aa82b9085d3fe050c7ca9d1392a7627e942"} Oct 31 13:49:06.395906 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:06.395774 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:49:06.396217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:06.396189 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 13:49:06.396360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:06.396338 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 13:49:06.797645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:06.796992 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:49:06.797645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:06.797127 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 13:49:06.797645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:06.797169 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 13:49:07.394004 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:07.393955 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 13:49:07.394532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:07.394022 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 13:49:07.787675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:07.787417 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 13:49:07.787675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:07.787477 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 13:49:08.397641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:08.395155 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 13:49:08.397641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:08.395224 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 13:49:08.788132 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:08.788028 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 13:49:08.788422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:08.788388 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 13:49:09.394932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:09.394893 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 13:49:09.395254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:09.395222 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 13:49:09.784677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:09.783944 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 13:49:09.784677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:09.784050 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 13:49:10.395236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:10.395155 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 13:49:10.395236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:10.395225 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 13:49:10.788676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:10.788120 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 13:49:10.788676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:10.788193 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 13:49:11.413620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:11.397875 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" start-of-body= Oct 31 13:49:11.413620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:11.397978 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.219:15021/healthz/ready\": dial tcp 10.132.0.219:15021: connect: connection refused" Oct 31 13:49:11.784684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:11.784214 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 31 13:49:11.784684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:11.784283 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 31 13:49:12.398856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.398821 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:49:12.399307 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.399176 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:49:12.400144 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.400121 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 13:49:12.439806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.439767 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x"] Oct 31 13:49:12.443540 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.443522 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" Oct 31 13:49:12.448761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.448730 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:49:12.448924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.448782 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:49:12.451944 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.451918 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x"] Oct 31 13:49:12.510031 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.509996 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1b7cb3-tmz2x\" (UID: \"8802788c-b651-4dbf-9281-98620e7946ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" Oct 31 13:49:12.510261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.510237 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95zhv\" (UniqueName: \"kubernetes.io/projected/8802788c-b651-4dbf-9281-98620e7946ae-kube-api-access-95zhv\") pod \"maas-api-revoke-sub-0f1b7cb3-tmz2x\" (UID: \"8802788c-b651-4dbf-9281-98620e7946ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" Oct 31 13:49:12.510374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.510316 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1b7cb3-tmz2x\" (UID: \"8802788c-b651-4dbf-9281-98620e7946ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" Oct 31 13:49:12.611772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.611728 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1b7cb3-tmz2x\" (UID: \"8802788c-b651-4dbf-9281-98620e7946ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" Oct 31 13:49:12.612002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.611984 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95zhv\" (UniqueName: \"kubernetes.io/projected/8802788c-b651-4dbf-9281-98620e7946ae-kube-api-access-95zhv\") pod \"maas-api-revoke-sub-0f1b7cb3-tmz2x\" (UID: \"8802788c-b651-4dbf-9281-98620e7946ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" Oct 31 13:49:12.612088 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.612072 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1b7cb3-tmz2x\" (UID: \"8802788c-b651-4dbf-9281-98620e7946ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" Oct 31 13:49:12.612770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.612746 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0f1b7cb3-tmz2x\" (UID: \"8802788c-b651-4dbf-9281-98620e7946ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" Oct 31 13:49:12.614327 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.614304 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0f1b7cb3-tmz2x\" (UID: \"8802788c-b651-4dbf-9281-98620e7946ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" Oct 31 13:49:12.626175 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.626151 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95zhv\" (UniqueName: \"kubernetes.io/projected/8802788c-b651-4dbf-9281-98620e7946ae-kube-api-access-95zhv\") pod \"maas-api-revoke-sub-0f1b7cb3-tmz2x\" (UID: \"8802788c-b651-4dbf-9281-98620e7946ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" Oct 31 13:49:12.764229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.764147 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" Oct 31 13:49:12.789525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.789494 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:49:12.790452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.790430 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:49:12.791263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:12.791244 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 13:49:13.055922 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:49:13.055891 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8802788c_b651_4dbf_9281_98620e7946ae.slice/crio-45ddc4a425d9cd8dd2539104cc3e49d8f191fc67cfa9bd9985a6edb049805814 WatchSource:0}: Error finding container 45ddc4a425d9cd8dd2539104cc3e49d8f191fc67cfa9bd9985a6edb049805814: Status 404 returned error can't find the container with id 45ddc4a425d9cd8dd2539104cc3e49d8f191fc67cfa9bd9985a6edb049805814 Oct 31 13:49:13.056032 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:13.055964 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x"] Oct 31 13:49:13.317214 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:13.317119 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" event={"ID":"8802788c-b651-4dbf-9281-98620e7946ae","Type":"ContainerStarted","Data":"45ddc4a425d9cd8dd2539104cc3e49d8f191fc67cfa9bd9985a6edb049805814"} Oct 31 13:49:14.331083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:14.330979 5532 generic.go:356] "Generic (PLEG): container finished" podID="8802788c-b651-4dbf-9281-98620e7946ae" containerID="4a3dcfe9b694f73e55d5eff26c0a59054f5eba4e7366ef7a620f67baf0937787" exitCode=0 Oct 31 13:49:14.331083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:14.331063 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" event={"ID":"8802788c-b651-4dbf-9281-98620e7946ae","Type":"ContainerDied","Data":"4a3dcfe9b694f73e55d5eff26c0a59054f5eba4e7366ef7a620f67baf0937787"} Oct 31 13:49:15.529211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:15.529182 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" Oct 31 13:49:15.562139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:15.562107 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-openshift-ca\") pod \"8802788c-b651-4dbf-9281-98620e7946ae\" (UID: \"8802788c-b651-4dbf-9281-98620e7946ae\") " Oct 31 13:49:15.562348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:15.562257 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-xks-ca\") pod \"8802788c-b651-4dbf-9281-98620e7946ae\" (UID: \"8802788c-b651-4dbf-9281-98620e7946ae\") " Oct 31 13:49:15.562348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:15.562300 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8802788c-b651-4dbf-9281-98620e7946ae-kube-api-access-95zhv\" (UniqueName: \"kubernetes.io/projected/8802788c-b651-4dbf-9281-98620e7946ae-kube-api-access-95zhv\") pod \"8802788c-b651-4dbf-9281-98620e7946ae\" (UID: \"8802788c-b651-4dbf-9281-98620e7946ae\") " Oct 31 13:49:15.562532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:15.562506 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-openshift-ca" pod "8802788c-b651-4dbf-9281-98620e7946ae" (UID: "8802788c-b651-4dbf-9281-98620e7946ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:49:15.563095 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:15.563066 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:15.564545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:15.564523 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-xks-ca" pod "8802788c-b651-4dbf-9281-98620e7946ae" (UID: "8802788c-b651-4dbf-9281-98620e7946ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:49:15.564710 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:15.564662 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8802788c-b651-4dbf-9281-98620e7946ae-kube-api-access-95zhv" pod "8802788c-b651-4dbf-9281-98620e7946ae" (UID: "8802788c-b651-4dbf-9281-98620e7946ae"). InnerVolumeSpecName "kube-api-access-95zhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:49:15.664360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:15.664326 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8802788c-b651-4dbf-9281-98620e7946ae-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:15.664555 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:15.664378 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95zhv\" (UniqueName: \"kubernetes.io/projected/8802788c-b651-4dbf-9281-98620e7946ae-kube-api-access-95zhv\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:16.348101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:16.348023 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" Oct 31 13:49:16.348101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:16.348052 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x" event={"ID":"8802788c-b651-4dbf-9281-98620e7946ae","Type":"ContainerDied","Data":"45ddc4a425d9cd8dd2539104cc3e49d8f191fc67cfa9bd9985a6edb049805814"} Oct 31 13:49:16.348290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:16.348103 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45ddc4a425d9cd8dd2539104cc3e49d8f191fc67cfa9bd9985a6edb049805814" Oct 31 13:49:20.817061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.817029 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn"] Oct 31 13:49:20.817882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.817860 5532 cpu_manager.go:414] "containerMap: removing container" podUID="8802788c-b651-4dbf-9281-98620e7946ae" containerName="revoke-keys" Oct 31 13:49:20.817882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.817879 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8802788c-b651-4dbf-9281-98620e7946ae" containerName="revoke-keys" Oct 31 13:49:20.818023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.818018 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="8802788c-b651-4dbf-9281-98620e7946ae" containerName="revoke-keys" Oct 31 13:49:20.821659 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.821639 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:20.836393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.836366 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-35d13e\"/\"kube-root-ca.crt\"" Oct 31 13:49:20.837964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.837941 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-35d13e\"/\"openshift-service-ca.crt\"" Oct 31 13:49:20.837964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.837953 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-35d13e\"/\"auth-test-model-35d13e-kserve-self-signed-certs\"" Oct 31 13:49:20.838121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.838096 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-35d13e\"/\"default-dockercfg-fnmmt\"" Oct 31 13:49:20.842013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.841993 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn"] Oct 31 13:49:20.948249 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.948219 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tls-certs\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:20.948430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.948380 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d7fk\" (UniqueName: \"kubernetes.io/projected/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-kube-api-access-5d7fk\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:20.948477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.948447 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-model-cache\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:20.948580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.948561 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-home\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:20.948659 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.948604 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tmp-dir\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:20.948702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:20.948659 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-dshm\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.051926 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.051878 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tls-certs\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.052133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.052034 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d7fk\" (UniqueName: \"kubernetes.io/projected/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-kube-api-access-5d7fk\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.052133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.052108 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-model-cache\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.052271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.052198 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-home\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.052271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.052230 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tmp-dir\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.052377 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.052278 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-dshm\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.053565 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.052971 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-home\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.054354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.054202 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tmp-dir\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.058492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.058458 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-model-cache\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.059375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.059354 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-dshm\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.062808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.062783 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tls-certs\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.067185 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.067134 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d7fk\" (UniqueName: \"kubernetes.io/projected/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-kube-api-access-5d7fk\") pod \"auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.143637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.143603 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:21.458151 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:21.457633 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn"] Oct 31 13:49:21.458354 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:49:21.458325 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c2fd3e5_38a5_4872_9ae8_9856d24cc3f8.slice/crio-75da5561d501b6a2fec65e5178c4dfff14813c670c52117c058b1cfa3cd61d13 WatchSource:0}: Error finding container 75da5561d501b6a2fec65e5178c4dfff14813c670c52117c058b1cfa3cd61d13: Status 404 returned error can't find the container with id 75da5561d501b6a2fec65e5178c4dfff14813c670c52117c058b1cfa3cd61d13 Oct 31 13:49:22.435925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:22.435888 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" event={"ID":"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8","Type":"ContainerStarted","Data":"95b0e220e341523aa6f97af00e31e396e6fbf7ea7d271bbf3326f9e139c41ff0"} Oct 31 13:49:22.436356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:22.435935 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" event={"ID":"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8","Type":"ContainerStarted","Data":"75da5561d501b6a2fec65e5178c4dfff14813c670c52117c058b1cfa3cd61d13"} Oct 31 13:49:22.466491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:22.466434 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" podStartSLOduration=2.466419834 podStartE2EDuration="2.466419834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:49:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:49:22.463075443 +0000 UTC m=+2730.608819270" watchObservedRunningTime="2026-10-31 13:49:22.466419834 +0000 UTC m=+2730.612163715" Oct 31 13:49:23.144275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:23.144238 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:31.144795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:31.144754 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:31.157284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:31.157257 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:31.532896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:31.532810 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:49:31.767066 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:49:31.767025 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:49:34.321677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.321647 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986"] Oct 31 13:49:34.326563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.326542 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" Oct 31 13:49:34.336884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.336853 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:49:34.337075 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.337054 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:49:34.339852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.339807 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986"] Oct 31 13:49:34.377503 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.377464 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpnh2\" (UniqueName: \"kubernetes.io/projected/e7acd287-0ae9-472a-a662-88da13fd673a-kube-api-access-wpnh2\") pod \"maas-api-revoke-sub-fd7a526a-4g986\" (UID: \"e7acd287-0ae9-472a-a662-88da13fd673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" Oct 31 13:49:34.377714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.377527 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7a526a-4g986\" (UID: \"e7acd287-0ae9-472a-a662-88da13fd673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" Oct 31 13:49:34.377880 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.377855 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7a526a-4g986\" (UID: \"e7acd287-0ae9-472a-a662-88da13fd673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" Oct 31 13:49:34.484974 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.484925 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7a526a-4g986\" (UID: \"e7acd287-0ae9-472a-a662-88da13fd673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" Oct 31 13:49:34.485452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.485426 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpnh2\" (UniqueName: \"kubernetes.io/projected/e7acd287-0ae9-472a-a662-88da13fd673a-kube-api-access-wpnh2\") pod \"maas-api-revoke-sub-fd7a526a-4g986\" (UID: \"e7acd287-0ae9-472a-a662-88da13fd673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" Oct 31 13:49:34.485648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.485629 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7a526a-4g986\" (UID: \"e7acd287-0ae9-472a-a662-88da13fd673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" Oct 31 13:49:34.489461 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.489436 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7a526a-4g986\" (UID: \"e7acd287-0ae9-472a-a662-88da13fd673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" Oct 31 13:49:34.491779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.491751 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7a526a-4g986\" (UID: \"e7acd287-0ae9-472a-a662-88da13fd673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" Oct 31 13:49:34.507144 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.507119 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpnh2\" (UniqueName: \"kubernetes.io/projected/e7acd287-0ae9-472a-a662-88da13fd673a-kube-api-access-wpnh2\") pod \"maas-api-revoke-sub-fd7a526a-4g986\" (UID: \"e7acd287-0ae9-472a-a662-88da13fd673a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" Oct 31 13:49:34.645828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:34.645801 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" Oct 31 13:49:35.022186 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:49:35.015526 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7acd287_0ae9_472a_a662_88da13fd673a.slice/crio-c23b62b0779049c8ea02b9a249dc3611c03d6e5a339e0d44920e04689c82fa2b WatchSource:0}: Error finding container c23b62b0779049c8ea02b9a249dc3611c03d6e5a339e0d44920e04689c82fa2b: Status 404 returned error can't find the container with id c23b62b0779049c8ea02b9a249dc3611c03d6e5a339e0d44920e04689c82fa2b Oct 31 13:49:35.022186 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:35.022112 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986"] Oct 31 13:49:35.561074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:35.560970 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" event={"ID":"e7acd287-0ae9-472a-a662-88da13fd673a","Type":"ContainerStarted","Data":"2641a177e6570c7ed8fc82e0436ac3cb1942c627a4af2b1f76df158e9e45ea89"} Oct 31 13:49:35.561074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:35.561056 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" event={"ID":"e7acd287-0ae9-472a-a662-88da13fd673a","Type":"ContainerStarted","Data":"c23b62b0779049c8ea02b9a249dc3611c03d6e5a339e0d44920e04689c82fa2b"} Oct 31 13:49:35.588207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:35.588140 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" podStartSLOduration=1.588121738 podStartE2EDuration="1.588121738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:49:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:49:35.582693717 +0000 UTC m=+2743.728437545" watchObservedRunningTime="2026-10-31 13:49:35.588121738 +0000 UTC m=+2743.733865567" Oct 31 13:49:36.573154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.573115 5532 generic.go:356] "Generic (PLEG): container finished" podID="e7acd287-0ae9-472a-a662-88da13fd673a" containerID="2641a177e6570c7ed8fc82e0436ac3cb1942c627a4af2b1f76df158e9e45ea89" exitCode=0 Oct 31 13:49:36.573568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.573156 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" event={"ID":"e7acd287-0ae9-472a-a662-88da13fd673a","Type":"ContainerDied","Data":"2641a177e6570c7ed8fc82e0436ac3cb1942c627a4af2b1f76df158e9e45ea89"} Oct 31 13:49:36.646779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.646745 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw"] Oct 31 13:49:36.651785 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.651764 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.657810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.657780 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-14ba5a\"/\"openshift-service-ca.crt\"" Oct 31 13:49:36.659214 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.659034 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-14ba5a\"/\"kube-root-ca.crt\"" Oct 31 13:49:36.659901 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.659842 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-14ba5a\"/\"auth-test-model-14ba5a-kserve-self-signed-certs\"" Oct 31 13:49:36.660091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.660071 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-14ba5a\"/\"default-dockercfg-ztxfb\"" Oct 31 13:49:36.663674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.663651 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw"] Oct 31 13:49:36.733202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.733168 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-home\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.733366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.733237 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tls-certs\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.733366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.733349 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tmp-dir\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.733539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.733522 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-model-cache\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.733682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.733641 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxz9f\" (UniqueName: \"kubernetes.io/projected/2257c6d8-8a08-4d03-9bad-48dd3493aecd-kube-api-access-qxz9f\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.733747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.733727 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-dshm\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.860671 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.859034 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxz9f\" (UniqueName: \"kubernetes.io/projected/2257c6d8-8a08-4d03-9bad-48dd3493aecd-kube-api-access-qxz9f\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.860671 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.859151 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-dshm\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.860671 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.859224 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-home\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.860671 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.859323 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tls-certs\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.860671 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.859395 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tmp-dir\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.860671 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.859667 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-model-cache\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.860671 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.860264 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-model-cache\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.861191 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.860909 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-home\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.871626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.861588 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tmp-dir\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.881243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.881197 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tls-certs\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.883961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.883930 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-dshm\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.900409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.897323 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxz9f\" (UniqueName: \"kubernetes.io/projected/2257c6d8-8a08-4d03-9bad-48dd3493aecd-kube-api-access-qxz9f\") pod \"auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:36.967482 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:36.967444 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:37.264953 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:49:37.264919 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2257c6d8_8a08_4d03_9bad_48dd3493aecd.slice/crio-ea7f9077afdd8eba8474af3ff0f8dc1efb62cf2d276de0c68e2aa50613f46ace WatchSource:0}: Error finding container ea7f9077afdd8eba8474af3ff0f8dc1efb62cf2d276de0c68e2aa50613f46ace: Status 404 returned error can't find the container with id ea7f9077afdd8eba8474af3ff0f8dc1efb62cf2d276de0c68e2aa50613f46ace Oct 31 13:49:37.265103 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.265047 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw"] Oct 31 13:49:37.583450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.583358 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" event={"ID":"2257c6d8-8a08-4d03-9bad-48dd3493aecd","Type":"ContainerStarted","Data":"bb5322994802242b1b5e2f833cce8d3d817fc084a04541331b5d89acf7c8a3c1"} Oct 31 13:49:37.583450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.583405 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" event={"ID":"2257c6d8-8a08-4d03-9bad-48dd3493aecd","Type":"ContainerStarted","Data":"ea7f9077afdd8eba8474af3ff0f8dc1efb62cf2d276de0c68e2aa50613f46ace"} Oct 31 13:49:37.605843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.605796 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" podStartSLOduration=1.6057824520000001 podStartE2EDuration="1.605782452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:49:37.603741403 +0000 UTC m=+2745.749485245" watchObservedRunningTime="2026-10-31 13:49:37.605782452 +0000 UTC m=+2745.751526280" Oct 31 13:49:37.731815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.731754 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" Oct 31 13:49:37.875820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.875740 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-openshift-ca\") pod \"e7acd287-0ae9-472a-a662-88da13fd673a\" (UID: \"e7acd287-0ae9-472a-a662-88da13fd673a\") " Oct 31 13:49:37.876008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.875820 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7acd287-0ae9-472a-a662-88da13fd673a-kube-api-access-wpnh2\" (UniqueName: \"kubernetes.io/projected/e7acd287-0ae9-472a-a662-88da13fd673a-kube-api-access-wpnh2\") pod \"e7acd287-0ae9-472a-a662-88da13fd673a\" (UID: \"e7acd287-0ae9-472a-a662-88da13fd673a\") " Oct 31 13:49:37.876008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.875849 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-xks-ca\") pod \"e7acd287-0ae9-472a-a662-88da13fd673a\" (UID: \"e7acd287-0ae9-472a-a662-88da13fd673a\") " Oct 31 13:49:37.876144 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.876094 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-openshift-ca" pod "e7acd287-0ae9-472a-a662-88da13fd673a" (UID: "e7acd287-0ae9-472a-a662-88da13fd673a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:49:37.876678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.876653 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:37.878081 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.878052 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7acd287-0ae9-472a-a662-88da13fd673a-kube-api-access-wpnh2" pod "e7acd287-0ae9-472a-a662-88da13fd673a" (UID: "e7acd287-0ae9-472a-a662-88da13fd673a"). InnerVolumeSpecName "kube-api-access-wpnh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:49:37.878193 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.878091 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-xks-ca" pod "e7acd287-0ae9-472a-a662-88da13fd673a" (UID: "e7acd287-0ae9-472a-a662-88da13fd673a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:49:37.968144 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.968113 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:37.978332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.978302 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpnh2\" (UniqueName: \"kubernetes.io/projected/e7acd287-0ae9-472a-a662-88da13fd673a-kube-api-access-wpnh2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:37.978332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:37.978328 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7acd287-0ae9-472a-a662-88da13fd673a-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:38.593498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:38.593469 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" Oct 31 13:49:38.593992 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:38.593503 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986" event={"ID":"e7acd287-0ae9-472a-a662-88da13fd673a","Type":"ContainerDied","Data":"c23b62b0779049c8ea02b9a249dc3611c03d6e5a339e0d44920e04689c82fa2b"} Oct 31 13:49:38.593992 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:38.593550 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c23b62b0779049c8ea02b9a249dc3611c03d6e5a339e0d44920e04689c82fa2b" Oct 31 13:49:46.967620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:46.967546 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:46.980917 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:46.980886 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:47.699989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:47.699951 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:49:48.936607 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:48.936555 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:49:49.032355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:49.032322 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh"] Oct 31 13:49:49.036276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:49.036253 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd7417ec-v6qbh"] Oct 31 13:49:49.975722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:49.975687 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h"] Oct 31 13:49:49.976490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:49.976471 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e7acd287-0ae9-472a-a662-88da13fd673a" containerName="revoke-keys" Oct 31 13:49:49.976490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:49.976486 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7acd287-0ae9-472a-a662-88da13fd673a" containerName="revoke-keys" Oct 31 13:49:49.976709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:49.976672 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7acd287-0ae9-472a-a662-88da13fd673a" containerName="revoke-keys" Oct 31 13:49:49.980688 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:49.980668 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" Oct 31 13:49:49.986382 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:49.986355 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:49:49.986538 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:49.986416 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:49:50.012068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:49.990842 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h"] Oct 31 13:49:50.022502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:50.022466 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ba2749d-8c41-4622-abe7-33cd7ed5aef5" path="/var/lib/kubelet/pods/8ba2749d-8c41-4622-abe7-33cd7ed5aef5/volumes" Oct 31 13:49:50.094097 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:50.094059 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-554759-mnh8h\" (UID: \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" Oct 31 13:49:50.094287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:50.094208 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-554759-mnh8h\" (UID: \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" Oct 31 13:49:50.094287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:50.094234 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhcfd\" (UniqueName: \"kubernetes.io/projected/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-kube-api-access-nhcfd\") pod \"maas-api-revoke-keys-e2e-inf-a-554759-mnh8h\" (UID: \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" Oct 31 13:49:50.195961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:50.195926 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-554759-mnh8h\" (UID: \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" Oct 31 13:49:50.196146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:50.196043 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-554759-mnh8h\" (UID: \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" Oct 31 13:49:50.196146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:50.196069 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhcfd\" (UniqueName: \"kubernetes.io/projected/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-kube-api-access-nhcfd\") pod \"maas-api-revoke-keys-e2e-inf-a-554759-mnh8h\" (UID: \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" Oct 31 13:49:50.196715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:50.196687 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-554759-mnh8h\" (UID: \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" Oct 31 13:49:50.198442 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:50.198417 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-554759-mnh8h\" (UID: \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" Oct 31 13:49:50.209934 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:50.209900 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhcfd\" (UniqueName: \"kubernetes.io/projected/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-kube-api-access-nhcfd\") pod \"maas-api-revoke-keys-e2e-inf-a-554759-mnh8h\" (UID: \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" Oct 31 13:49:50.321159 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:50.321075 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" Oct 31 13:49:50.632155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:50.630970 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h"] Oct 31 13:49:50.635473 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:49:50.635420 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ea5f3e_5c9c_482e_853b_276c4a76b4c5.slice/crio-682452c640b9bd97d026cc4c92eafaecd387626343be13749c641a98db811328 WatchSource:0}: Error finding container 682452c640b9bd97d026cc4c92eafaecd387626343be13749c641a98db811328: Status 404 returned error can't find the container with id 682452c640b9bd97d026cc4c92eafaecd387626343be13749c641a98db811328 Oct 31 13:49:50.722939 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:50.722908 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" event={"ID":"81ea5f3e-5c9c-482e-853b-276c4a76b4c5","Type":"ContainerStarted","Data":"682452c640b9bd97d026cc4c92eafaecd387626343be13749c641a98db811328"} Oct 31 13:49:51.777674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:51.777622 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" event={"ID":"81ea5f3e-5c9c-482e-853b-276c4a76b4c5","Type":"ContainerStarted","Data":"5f60c1e137fa099b942ed63c7e63ec245d002225a91c58c65af6961904871147"} Oct 31 13:49:51.808315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:51.806489 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" podStartSLOduration=2.806471209 podStartE2EDuration="2.806471209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:49:51.805709951 +0000 UTC m=+2759.951453780" watchObservedRunningTime="2026-10-31 13:49:51.806471209 +0000 UTC m=+2759.952215039" Oct 31 13:49:52.787863 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:52.787825 5532 generic.go:356] "Generic (PLEG): container finished" podID="81ea5f3e-5c9c-482e-853b-276c4a76b4c5" containerID="5f60c1e137fa099b942ed63c7e63ec245d002225a91c58c65af6961904871147" exitCode=0 Oct 31 13:49:52.788297 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:52.787908 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" event={"ID":"81ea5f3e-5c9c-482e-853b-276c4a76b4c5","Type":"ContainerDied","Data":"5f60c1e137fa099b942ed63c7e63ec245d002225a91c58c65af6961904871147"} Oct 31 13:49:53.970156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:53.970130 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" Oct 31 13:49:54.096093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:54.096017 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-kube-api-access-nhcfd\" (UniqueName: \"kubernetes.io/projected/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-kube-api-access-nhcfd\") pod \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\" (UID: \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\") " Oct 31 13:49:54.096276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:54.096125 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-xks-ca\") pod \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\" (UID: \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\") " Oct 31 13:49:54.096397 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:54.096355 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-openshift-ca\") pod \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\" (UID: \"81ea5f3e-5c9c-482e-853b-276c4a76b4c5\") " Oct 31 13:49:54.096713 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:54.096682 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-openshift-ca" pod "81ea5f3e-5c9c-482e-853b-276c4a76b4c5" (UID: "81ea5f3e-5c9c-482e-853b-276c4a76b4c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:49:54.097189 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:54.097167 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:54.098518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:54.098491 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-xks-ca" pod "81ea5f3e-5c9c-482e-853b-276c4a76b4c5" (UID: "81ea5f3e-5c9c-482e-853b-276c4a76b4c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:49:54.098676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:54.098548 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-kube-api-access-nhcfd" pod "81ea5f3e-5c9c-482e-853b-276c4a76b4c5" (UID: "81ea5f3e-5c9c-482e-853b-276c4a76b4c5"). InnerVolumeSpecName "kube-api-access-nhcfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:49:54.198429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:54.198395 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhcfd\" (UniqueName: \"kubernetes.io/projected/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-kube-api-access-nhcfd\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:54.198429 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:54.198423 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81ea5f3e-5c9c-482e-853b-276c4a76b4c5-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:49:54.805240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:54.805201 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" Oct 31 13:49:54.805240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:54.805233 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h" event={"ID":"81ea5f3e-5c9c-482e-853b-276c4a76b4c5","Type":"ContainerDied","Data":"682452c640b9bd97d026cc4c92eafaecd387626343be13749c641a98db811328"} Oct 31 13:49:54.805506 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:49:54.805280 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="682452c640b9bd97d026cc4c92eafaecd387626343be13749c641a98db811328" Oct 31 13:50:00.023397 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:00.023352 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd"] Oct 31 13:50:00.031805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:00.031774 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7"] Oct 31 13:50:00.036451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:00.036426 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d74d07cc-vv9n7"] Oct 31 13:50:01.763286 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:50:01.763249 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:50:02.006170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:02.006136 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1f8b08-ffe4-4239-8572-bbafda1a2dde" path="/var/lib/kubelet/pods/6a1f8b08-ffe4-4239-8572-bbafda1a2dde/volumes" Oct 31 13:50:05.023941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:05.023891 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" podUID="ed127dcb-547e-44e7-af88-c7b6ee439b81" containerName="payload-processing" containerID="cri-o://daf139136f7562240ea09369447cda0bb53cf17bc68ce4a026fe739e9a41a737" gracePeriod=25 Oct 31 13:50:12.597028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:12.596990 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" podUID="ed127dcb-547e-44e7-af88-c7b6ee439b81" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:50:14.340781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:14.340745 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h"] Oct 31 13:50:14.347942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:14.347910 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-554759-mnh8h"] Oct 31 13:50:16.006628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:16.006538 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81ea5f3e-5c9c-482e-853b-276c4a76b4c5" path="/var/lib/kubelet/pods/81ea5f3e-5c9c-482e-853b-276c4a76b4c5/volumes" Oct 31 13:50:22.488062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.488029 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9"] Oct 31 13:50:22.488875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.488856 5532 cpu_manager.go:414] "containerMap: removing container" podUID="81ea5f3e-5c9c-482e-853b-276c4a76b4c5" containerName="revoke-keys" Oct 31 13:50:22.488953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.488875 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81ea5f3e-5c9c-482e-853b-276c4a76b4c5" containerName="revoke-keys" Oct 31 13:50:22.489045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.489036 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="81ea5f3e-5c9c-482e-853b-276c4a76b4c5" containerName="revoke-keys" Oct 31 13:50:22.492744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.492727 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" Oct 31 13:50:22.501989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.501966 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9"] Oct 31 13:50:22.569526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.569494 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9\" (UID: \"bc5f1718-e2f6-4388-b571-f3240a01b5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" Oct 31 13:50:22.569756 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.569735 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrxq2\" (UniqueName: \"kubernetes.io/projected/bc5f1718-e2f6-4388-b571-f3240a01b5da-kube-api-access-qrxq2\") pod \"maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9\" (UID: \"bc5f1718-e2f6-4388-b571-f3240a01b5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" Oct 31 13:50:22.569845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.569829 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9\" (UID: \"bc5f1718-e2f6-4388-b571-f3240a01b5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" Oct 31 13:50:22.576970 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.576951 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:50:22.577068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.577052 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:50:22.596716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.596686 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" podUID="ed127dcb-547e-44e7-af88-c7b6ee439b81" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:50:22.671813 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.671781 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9\" (UID: \"bc5f1718-e2f6-4388-b571-f3240a01b5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" Oct 31 13:50:22.671978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.671966 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrxq2\" (UniqueName: \"kubernetes.io/projected/bc5f1718-e2f6-4388-b571-f3240a01b5da-kube-api-access-qrxq2\") pod \"maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9\" (UID: \"bc5f1718-e2f6-4388-b571-f3240a01b5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" Oct 31 13:50:22.672054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.672034 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9\" (UID: \"bc5f1718-e2f6-4388-b571-f3240a01b5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" Oct 31 13:50:22.672448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.672422 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9\" (UID: \"bc5f1718-e2f6-4388-b571-f3240a01b5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" Oct 31 13:50:22.674214 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.674195 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9\" (UID: \"bc5f1718-e2f6-4388-b571-f3240a01b5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" Oct 31 13:50:22.685270 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.685248 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrxq2\" (UniqueName: \"kubernetes.io/projected/bc5f1718-e2f6-4388-b571-f3240a01b5da-kube-api-access-qrxq2\") pod \"maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9\" (UID: \"bc5f1718-e2f6-4388-b571-f3240a01b5da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" Oct 31 13:50:22.809649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:22.809547 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" Oct 31 13:50:23.099223 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:50:23.099196 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc5f1718_e2f6_4388_b571_f3240a01b5da.slice/crio-6d5fc19c5cd2464805ff8641527bb877584e36895d835ed657ba854a4c6526ce WatchSource:0}: Error finding container 6d5fc19c5cd2464805ff8641527bb877584e36895d835ed657ba854a4c6526ce: Status 404 returned error can't find the container with id 6d5fc19c5cd2464805ff8641527bb877584e36895d835ed657ba854a4c6526ce Oct 31 13:50:23.100127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:23.100103 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9"] Oct 31 13:50:24.029445 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.029406 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w"] Oct 31 13:50:24.033239 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.033212 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67a40b26-r2n7w"] Oct 31 13:50:24.113644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.113572 5532 generic.go:356] "Generic (PLEG): container finished" podID="bc5f1718-e2f6-4388-b571-f3240a01b5da" containerID="0206082d7ef3b4aebcdcdde880648e37c3d18b1f19263e9246bff8b1e308a879" exitCode=0 Oct 31 13:50:24.113862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.113796 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" event={"ID":"bc5f1718-e2f6-4388-b571-f3240a01b5da","Type":"ContainerDied","Data":"0206082d7ef3b4aebcdcdde880648e37c3d18b1f19263e9246bff8b1e308a879"} Oct 31 13:50:24.113862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.113845 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" event={"ID":"bc5f1718-e2f6-4388-b571-f3240a01b5da","Type":"ContainerStarted","Data":"6d5fc19c5cd2464805ff8641527bb877584e36895d835ed657ba854a4c6526ce"} Oct 31 13:50:24.155208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.155179 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt"] Oct 31 13:50:24.160325 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.160302 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.168638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.168614 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-599291-openshift-default-dockercfg-9db8q\"" Oct 31 13:50:24.173773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.173748 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt"] Oct 31 13:50:24.311519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.311436 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-istio-token\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.311728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.311618 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-envoy\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.311728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.311648 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-data\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.311728 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.311680 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279335a6-fc12-417b-824a-9ceafe2d3431-istio-podinfo\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.311878 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.311833 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-certs\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.311878 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.311857 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccz4b\" (UniqueName: \"kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-kube-api-access-ccz4b\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.311991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.311967 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-credential-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.312113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.312098 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279335a6-fc12-417b-824a-9ceafe2d3431-istiod-ca-cert\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.312178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.312146 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.413568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.413536 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-credential-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.413754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.413651 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279335a6-fc12-417b-824a-9ceafe2d3431-istiod-ca-cert\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.413754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.413685 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.413832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.413764 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-istio-token\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.413928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.413910 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-envoy\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.413987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.413933 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-credential-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.413987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.413941 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-data\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.414091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.414022 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279335a6-fc12-417b-824a-9ceafe2d3431-istio-podinfo\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.414151 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.414090 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-socket\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.414216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.414177 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-certs\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.414285 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.414211 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccz4b\" (UniqueName: \"kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-kube-api-access-ccz4b\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.414345 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.414306 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-data\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.414448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.414426 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279335a6-fc12-417b-824a-9ceafe2d3431-istiod-ca-cert\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.414541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.414525 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-certs\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.416275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.416251 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-envoy\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.416416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.416398 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279335a6-fc12-417b-824a-9ceafe2d3431-istio-podinfo\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.429131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.429110 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-istio-token\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.429243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.429169 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccz4b\" (UniqueName: \"kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-kube-api-access-ccz4b\") pod \"e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.477282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.477245 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:24.777490 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.777463 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt"] Oct 31 13:50:24.778226 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:50:24.778196 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279335a6_fc12_417b_824a_9ceafe2d3431.slice/crio-27d8c486811e5751b856298e7c64e24075b059f9e82731bd305c999b6c4a8edb WatchSource:0}: Error finding container 27d8c486811e5751b856298e7c64e24075b059f9e82731bd305c999b6c4a8edb: Status 404 returned error can't find the container with id 27d8c486811e5751b856298e7c64e24075b059f9e82731bd305c999b6c4a8edb Oct 31 13:50:24.781314 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.781283 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:50:24.781405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.781348 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:50:24.781405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:24.781377 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:50:25.147148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.147110 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" event={"ID":"279335a6-fc12-417b-824a-9ceafe2d3431","Type":"ContainerStarted","Data":"b88d1347fe3624c93559a0aa42e61a11093681c0d9205b5349e3c754be09e11c"} Oct 31 13:50:25.149170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.147206 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" event={"ID":"279335a6-fc12-417b-824a-9ceafe2d3431","Type":"ContainerStarted","Data":"27d8c486811e5751b856298e7c64e24075b059f9e82731bd305c999b6c4a8edb"} Oct 31 13:50:25.180427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.180358 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" podStartSLOduration=1.180337191 podStartE2EDuration="1.180337191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:50:25.176220034 +0000 UTC m=+2793.321963862" watchObservedRunningTime="2026-10-31 13:50:25.180337191 +0000 UTC m=+2793.326081020" Oct 31 13:50:25.344322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.344294 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" Oct 31 13:50:25.378635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.377011 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" Oct 31 13:50:25.439650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.439578 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed127dcb-547e-44e7-af88-c7b6ee439b81-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed127dcb-547e-44e7-af88-c7b6ee439b81-plugins-config-volume\") pod \"ed127dcb-547e-44e7-af88-c7b6ee439b81\" (UID: \"ed127dcb-547e-44e7-af88-c7b6ee439b81\") " Oct 31 13:50:25.439852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.439712 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed127dcb-547e-44e7-af88-c7b6ee439b81-kube-api-access-klr22\" (UniqueName: \"kubernetes.io/projected/ed127dcb-547e-44e7-af88-c7b6ee439b81-kube-api-access-klr22\") pod \"ed127dcb-547e-44e7-af88-c7b6ee439b81\" (UID: \"ed127dcb-547e-44e7-af88-c7b6ee439b81\") " Oct 31 13:50:25.439957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.439936 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-xks-ca\") pod \"bc5f1718-e2f6-4388-b571-f3240a01b5da\" (UID: \"bc5f1718-e2f6-4388-b571-f3240a01b5da\") " Oct 31 13:50:25.440024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.439978 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed127dcb-547e-44e7-af88-c7b6ee439b81-plugins-config-volume" pod "ed127dcb-547e-44e7-af88-c7b6ee439b81" (UID: "ed127dcb-547e-44e7-af88-c7b6ee439b81"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:50:25.440024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.439985 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-openshift-ca\") pod \"bc5f1718-e2f6-4388-b571-f3240a01b5da\" (UID: \"bc5f1718-e2f6-4388-b571-f3240a01b5da\") " Oct 31 13:50:25.440143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.440119 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc5f1718-e2f6-4388-b571-f3240a01b5da-kube-api-access-qrxq2\" (UniqueName: \"kubernetes.io/projected/bc5f1718-e2f6-4388-b571-f3240a01b5da-kube-api-access-qrxq2\") pod \"bc5f1718-e2f6-4388-b571-f3240a01b5da\" (UID: \"bc5f1718-e2f6-4388-b571-f3240a01b5da\") " Oct 31 13:50:25.440426 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.440393 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-openshift-ca" pod "bc5f1718-e2f6-4388-b571-f3240a01b5da" (UID: "bc5f1718-e2f6-4388-b571-f3240a01b5da"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:50:25.441293 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.441265 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:25.441393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.441298 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed127dcb-547e-44e7-af88-c7b6ee439b81-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:25.443015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.442991 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc5f1718-e2f6-4388-b571-f3240a01b5da-kube-api-access-qrxq2" pod "bc5f1718-e2f6-4388-b571-f3240a01b5da" (UID: "bc5f1718-e2f6-4388-b571-f3240a01b5da"). InnerVolumeSpecName "kube-api-access-qrxq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:50:25.443953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.443933 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed127dcb-547e-44e7-af88-c7b6ee439b81-kube-api-access-klr22" pod "ed127dcb-547e-44e7-af88-c7b6ee439b81" (UID: "ed127dcb-547e-44e7-af88-c7b6ee439b81"). InnerVolumeSpecName "kube-api-access-klr22". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:50:25.445396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.445366 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-xks-ca" pod "bc5f1718-e2f6-4388-b571-f3240a01b5da" (UID: "bc5f1718-e2f6-4388-b571-f3240a01b5da"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:50:25.477387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.477364 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:25.478573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.478550 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 13:50:25.478680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.478624 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" podUID="279335a6-fc12-417b-824a-9ceafe2d3431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 13:50:25.548861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.548709 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc5f1718-e2f6-4388-b571-f3240a01b5da-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:25.548861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.548747 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrxq2\" (UniqueName: \"kubernetes.io/projected/bc5f1718-e2f6-4388-b571-f3240a01b5da-kube-api-access-qrxq2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:25.548861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:25.548763 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klr22\" (UniqueName: \"kubernetes.io/projected/ed127dcb-547e-44e7-af88-c7b6ee439b81-kube-api-access-klr22\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:26.005975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.005898 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c65a8d-5c95-4356-954d-106740ce3181" path="/var/lib/kubelet/pods/b9c65a8d-5c95-4356-954d-106740ce3181/volumes" Oct 31 13:50:26.171238 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.171201 5532 generic.go:356] "Generic (PLEG): container finished" podID="ed127dcb-547e-44e7-af88-c7b6ee439b81" containerID="daf139136f7562240ea09369447cda0bb53cf17bc68ce4a026fe739e9a41a737" exitCode=0 Oct 31 13:50:26.171747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.171247 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" Oct 31 13:50:26.171747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.171284 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" event={"ID":"ed127dcb-547e-44e7-af88-c7b6ee439b81","Type":"ContainerDied","Data":"daf139136f7562240ea09369447cda0bb53cf17bc68ce4a026fe739e9a41a737"} Oct 31 13:50:26.171747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.171337 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd" event={"ID":"ed127dcb-547e-44e7-af88-c7b6ee439b81","Type":"ContainerDied","Data":"964e0954c25812bfe0d7bbe495e532eebaf0a2722c647fb525832d76298b6f22"} Oct 31 13:50:26.171747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.171367 5532 scope.go:122] "RemoveContainer" containerID="daf139136f7562240ea09369447cda0bb53cf17bc68ce4a026fe739e9a41a737" Oct 31 13:50:26.172999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.172985 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" Oct 31 13:50:26.173098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.173033 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9" event={"ID":"bc5f1718-e2f6-4388-b571-f3240a01b5da","Type":"ContainerDied","Data":"6d5fc19c5cd2464805ff8641527bb877584e36895d835ed657ba854a4c6526ce"} Oct 31 13:50:26.173148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.173093 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d5fc19c5cd2464805ff8641527bb877584e36895d835ed657ba854a4c6526ce" Oct 31 13:50:26.183462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.183433 5532 scope.go:122] "RemoveContainer" containerID="daf139136f7562240ea09369447cda0bb53cf17bc68ce4a026fe739e9a41a737" Oct 31 13:50:26.183758 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:50:26.183740 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daf139136f7562240ea09369447cda0bb53cf17bc68ce4a026fe739e9a41a737\": container with ID starting with daf139136f7562240ea09369447cda0bb53cf17bc68ce4a026fe739e9a41a737 not found: ID does not exist" containerID="daf139136f7562240ea09369447cda0bb53cf17bc68ce4a026fe739e9a41a737" Oct 31 13:50:26.183822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.183766 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daf139136f7562240ea09369447cda0bb53cf17bc68ce4a026fe739e9a41a737"} err="failed to get container status \"daf139136f7562240ea09369447cda0bb53cf17bc68ce4a026fe739e9a41a737\": rpc error: code = NotFound desc = could not find container \"daf139136f7562240ea09369447cda0bb53cf17bc68ce4a026fe739e9a41a737\": container with ID starting with daf139136f7562240ea09369447cda0bb53cf17bc68ce4a026fe739e9a41a737 not found: ID does not exist" Oct 31 13:50:26.199278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.199253 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd"] Oct 31 13:50:26.203316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.203291 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-554759-db5b8c5dc-mvxbd"] Oct 31 13:50:26.478366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.478330 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 13:50:26.478632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:26.478389 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" podUID="279335a6-fc12-417b-824a-9ceafe2d3431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 13:50:27.485617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:27.484761 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 13:50:27.485617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:27.484826 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" podUID="279335a6-fc12-417b-824a-9ceafe2d3431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 13:50:28.049618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:28.048699 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed127dcb-547e-44e7-af88-c7b6ee439b81" path="/var/lib/kubelet/pods/ed127dcb-547e-44e7-af88-c7b6ee439b81/volumes" Oct 31 13:50:28.486660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:28.481410 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 13:50:28.486660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:28.481490 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" podUID="279335a6-fc12-417b-824a-9ceafe2d3431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 13:50:29.478153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:29.478114 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 13:50:29.478369 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:29.478180 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" podUID="279335a6-fc12-417b-824a-9ceafe2d3431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 13:50:30.478007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:30.477970 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 13:50:30.478423 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:30.478057 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" podUID="279335a6-fc12-417b-824a-9ceafe2d3431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 13:50:31.478265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:31.478224 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" start-of-body= Oct 31 13:50:31.478808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:31.478299 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" podUID="279335a6-fc12-417b-824a-9ceafe2d3431" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.228:15021/healthz/ready\": dial tcp 10.132.0.228:15021: connect: connection refused" Oct 31 13:50:31.766616 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:50:31.766494 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:50:32.482698 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:32.482668 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:32.483175 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:32.483143 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:32.483853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:32.483836 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 13:50:32.530938 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:32.530909 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8"] Oct 31 13:50:32.552426 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:32.552384 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44"] Oct 31 13:50:37.533008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:37.532956 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" podUID="6a1bbc81-7ef4-497e-b4aa-e06b3db670f5" containerName="payload-processing" containerID="cri-o://ddeeceeac7253ba5ab475834d9721a9cc3b33ffbecf9788b7ad2b42fb21bcfb8" gracePeriod=25 Oct 31 13:50:37.554115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:37.554065 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" podUID="64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c" containerName="payload-pre-processing" containerID="cri-o://641c80cb5498a5d776bdcf2047ea39fb9f542950256064964f6a451047a139ed" gracePeriod=25 Oct 31 13:50:39.240270 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:39.240230 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" podUID="64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:50:39.240713 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:39.240232 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" podUID="6a1bbc81-7ef4-497e-b4aa-e06b3db670f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:50:44.578182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:44.578143 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng"] Oct 31 13:50:44.578702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:44.578423 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" podUID="50a7df40-0977-49e6-9a4a-1669621ee182" containerName="maas-api" containerID="cri-o://432e920b01dc2d2e607b123c6aafef3b8bcc23d9654d3d32f3c10e5dbebc47c0" gracePeriod=30 Oct 31 13:50:45.029654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.029337 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:50:45.144552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.144509 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9"] Oct 31 13:50:45.147104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.147077 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-metrics-certs\") pod \"50a7df40-0977-49e6-9a4a-1669621ee182\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " Oct 31 13:50:45.147271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.147131 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50a7df40-0977-49e6-9a4a-1669621ee182-kube-api-access-8thw6\" (UniqueName: \"kubernetes.io/projected/50a7df40-0977-49e6-9a4a-1669621ee182-kube-api-access-8thw6\") pod \"50a7df40-0977-49e6-9a4a-1669621ee182\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " Oct 31 13:50:45.147271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.147217 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50a7df40-0977-49e6-9a4a-1669621ee182-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a7df40-0977-49e6-9a4a-1669621ee182-openshift-service-ca\") pod \"50a7df40-0977-49e6-9a4a-1669621ee182\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " Oct 31 13:50:45.147391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.147342 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-maas-api-tls\") pod \"50a7df40-0977-49e6-9a4a-1669621ee182\" (UID: \"50a7df40-0977-49e6-9a4a-1669621ee182\") " Oct 31 13:50:45.147683 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.147656 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50a7df40-0977-49e6-9a4a-1669621ee182-openshift-service-ca" pod "50a7df40-0977-49e6-9a4a-1669621ee182" (UID: "50a7df40-0977-49e6-9a4a-1669621ee182"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:50:45.148058 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.148038 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/50a7df40-0977-49e6-9a4a-1669621ee182-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:45.149128 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.149101 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-35b584-mqfp9"] Oct 31 13:50:45.149350 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.149324 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-maas-api-tls" pod "50a7df40-0977-49e6-9a4a-1669621ee182" (UID: "50a7df40-0977-49e6-9a4a-1669621ee182"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:50:45.149440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.149364 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-metrics-certs" pod "50a7df40-0977-49e6-9a4a-1669621ee182" (UID: "50a7df40-0977-49e6-9a4a-1669621ee182"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:50:45.149547 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.149533 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50a7df40-0977-49e6-9a4a-1669621ee182-kube-api-access-8thw6" pod "50a7df40-0977-49e6-9a4a-1669621ee182" (UID: "50a7df40-0977-49e6-9a4a-1669621ee182"). InnerVolumeSpecName "kube-api-access-8thw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:50:45.249487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.249400 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:45.249487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.249428 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50a7df40-0977-49e6-9a4a-1669621ee182-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:45.249487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.249439 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8thw6\" (UniqueName: \"kubernetes.io/projected/50a7df40-0977-49e6-9a4a-1669621ee182-kube-api-access-8thw6\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:45.391531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.391495 5532 generic.go:356] "Generic (PLEG): container finished" podID="50a7df40-0977-49e6-9a4a-1669621ee182" containerID="432e920b01dc2d2e607b123c6aafef3b8bcc23d9654d3d32f3c10e5dbebc47c0" exitCode=0 Oct 31 13:50:45.391821 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.391538 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" event={"ID":"50a7df40-0977-49e6-9a4a-1669621ee182","Type":"ContainerDied","Data":"432e920b01dc2d2e607b123c6aafef3b8bcc23d9654d3d32f3c10e5dbebc47c0"} Oct 31 13:50:45.391924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.391863 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" event={"ID":"50a7df40-0977-49e6-9a4a-1669621ee182","Type":"ContainerDied","Data":"4b017d44d477afaf9113041cedad119cca891bff7efb43253e2f02ef2d359568"} Oct 31 13:50:45.391924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.391557 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng" Oct 31 13:50:45.391924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.391901 5532 scope.go:122] "RemoveContainer" containerID="432e920b01dc2d2e607b123c6aafef3b8bcc23d9654d3d32f3c10e5dbebc47c0" Oct 31 13:50:45.408557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.408538 5532 scope.go:122] "RemoveContainer" containerID="432e920b01dc2d2e607b123c6aafef3b8bcc23d9654d3d32f3c10e5dbebc47c0" Oct 31 13:50:45.408861 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:50:45.408841 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432e920b01dc2d2e607b123c6aafef3b8bcc23d9654d3d32f3c10e5dbebc47c0\": container with ID starting with 432e920b01dc2d2e607b123c6aafef3b8bcc23d9654d3d32f3c10e5dbebc47c0 not found: ID does not exist" containerID="432e920b01dc2d2e607b123c6aafef3b8bcc23d9654d3d32f3c10e5dbebc47c0" Oct 31 13:50:45.408945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.408867 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432e920b01dc2d2e607b123c6aafef3b8bcc23d9654d3d32f3c10e5dbebc47c0"} err="failed to get container status \"432e920b01dc2d2e607b123c6aafef3b8bcc23d9654d3d32f3c10e5dbebc47c0\": rpc error: code = NotFound desc = could not find container \"432e920b01dc2d2e607b123c6aafef3b8bcc23d9654d3d32f3c10e5dbebc47c0\": container with ID starting with 432e920b01dc2d2e607b123c6aafef3b8bcc23d9654d3d32f3c10e5dbebc47c0 not found: ID does not exist" Oct 31 13:50:45.436839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.436796 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng"] Oct 31 13:50:45.442009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:45.441979 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-35b584-6d8cbbdd74-g64ng"] Oct 31 13:50:46.018332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:46.018297 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50a7df40-0977-49e6-9a4a-1669621ee182" path="/var/lib/kubelet/pods/50a7df40-0977-49e6-9a4a-1669621ee182/volumes" Oct 31 13:50:46.018824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:46.018809 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc5f1718-e2f6-4388-b571-f3240a01b5da" path="/var/lib/kubelet/pods/bc5f1718-e2f6-4388-b571-f3240a01b5da/volumes" Oct 31 13:50:49.240403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:49.240366 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" podUID="6a1bbc81-7ef4-497e-b4aa-e06b3db670f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:50:49.240832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:49.240366 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" podUID="64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:50:54.930055 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:54.930024 5532 scope.go:122] "RemoveContainer" containerID="b6004adca5ca1534c6680b4d0b0ff32746de63cc1d76731a171ea9dccb443679" Oct 31 13:50:56.152576 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.152538 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6"] Oct 31 13:50:56.153342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.153327 5532 cpu_manager.go:414] "containerMap: removing container" podUID="50a7df40-0977-49e6-9a4a-1669621ee182" containerName="maas-api" Oct 31 13:50:56.153387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.153343 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50a7df40-0977-49e6-9a4a-1669621ee182" containerName="maas-api" Oct 31 13:50:56.153387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.153373 5532 cpu_manager.go:414] "containerMap: removing container" podUID="bc5f1718-e2f6-4388-b571-f3240a01b5da" containerName="revoke-keys" Oct 31 13:50:56.153387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.153378 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc5f1718-e2f6-4388-b571-f3240a01b5da" containerName="revoke-keys" Oct 31 13:50:56.153488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.153387 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ed127dcb-547e-44e7-af88-c7b6ee439b81" containerName="payload-processing" Oct 31 13:50:56.153488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.153392 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed127dcb-547e-44e7-af88-c7b6ee439b81" containerName="payload-processing" Oct 31 13:50:56.153557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.153518 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc5f1718-e2f6-4388-b571-f3240a01b5da" containerName="revoke-keys" Oct 31 13:50:56.153557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.153525 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed127dcb-547e-44e7-af88-c7b6ee439b81" containerName="payload-processing" Oct 31 13:50:56.153557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.153542 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="50a7df40-0977-49e6-9a4a-1669621ee182" containerName="maas-api" Oct 31 13:50:56.158947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.158928 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" Oct 31 13:50:56.165481 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.165453 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:50:56.165622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.165566 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:50:56.170088 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.170068 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6"] Oct 31 13:50:56.271847 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.271808 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsrgt\" (UniqueName: \"kubernetes.io/projected/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-kube-api-access-rsrgt\") pod \"maas-api-revoke-sub-516f84cd-x9lr6\" (UID: \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" Oct 31 13:50:56.272047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.272025 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-516f84cd-x9lr6\" (UID: \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" Oct 31 13:50:56.272283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.272263 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-516f84cd-x9lr6\" (UID: \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" Oct 31 13:50:56.373784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.373747 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-516f84cd-x9lr6\" (UID: \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" Oct 31 13:50:56.373978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.373846 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsrgt\" (UniqueName: \"kubernetes.io/projected/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-kube-api-access-rsrgt\") pod \"maas-api-revoke-sub-516f84cd-x9lr6\" (UID: \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" Oct 31 13:50:56.374060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.373989 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-516f84cd-x9lr6\" (UID: \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" Oct 31 13:50:56.374487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.374463 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-516f84cd-x9lr6\" (UID: \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" Oct 31 13:50:56.376396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.376374 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-516f84cd-x9lr6\" (UID: \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" Oct 31 13:50:56.387539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.387511 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsrgt\" (UniqueName: \"kubernetes.io/projected/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-kube-api-access-rsrgt\") pod \"maas-api-revoke-sub-516f84cd-x9lr6\" (UID: \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" Oct 31 13:50:56.475820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.475757 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" Oct 31 13:50:56.766052 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:50:56.766020 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod832fb8fd_1d03_46ee_bdb5_4bcaba74e1b2.slice/crio-73f341c5e976045b92c140431e9c391384b5baa5dcb13ffc8c84b3012cdd4ffa WatchSource:0}: Error finding container 73f341c5e976045b92c140431e9c391384b5baa5dcb13ffc8c84b3012cdd4ffa: Status 404 returned error can't find the container with id 73f341c5e976045b92c140431e9c391384b5baa5dcb13ffc8c84b3012cdd4ffa Oct 31 13:50:56.766874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:56.766847 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6"] Oct 31 13:50:57.512845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:57.512759 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" event={"ID":"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2","Type":"ContainerStarted","Data":"04f26a39638f311cb8299e54e5c0aff68361a84febbba34899f58303f0b96554"} Oct 31 13:50:57.512845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:57.512824 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" event={"ID":"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2","Type":"ContainerStarted","Data":"73f341c5e976045b92c140431e9c391384b5baa5dcb13ffc8c84b3012cdd4ffa"} Oct 31 13:50:57.538147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:57.538098 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" podStartSLOduration=1.538084615 podStartE2EDuration="1.538084615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:50:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:50:57.533458679 +0000 UTC m=+2825.679202508" watchObservedRunningTime="2026-10-31 13:50:57.538084615 +0000 UTC m=+2825.683828466" Oct 31 13:50:57.859822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:57.859790 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" Oct 31 13:50:57.876513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:57.876487 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" Oct 31 13:50:58.021628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.021521 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-plugins-config-volume\") pod \"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5\" (UID: \"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5\") " Oct 31 13:50:58.021628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.021554 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-kube-api-access-jwsnm\" (UniqueName: \"kubernetes.io/projected/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-kube-api-access-jwsnm\") pod \"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5\" (UID: \"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5\") " Oct 31 13:50:58.021894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.021872 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-plugins-config-volume\") pod \"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c\" (UID: \"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c\") " Oct 31 13:50:58.021949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.021933 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-kube-api-access-66tht\" (UniqueName: \"kubernetes.io/projected/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-kube-api-access-66tht\") pod \"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c\" (UID: \"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c\") " Oct 31 13:50:58.021949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.021930 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-plugins-config-volume" pod "6a1bbc81-7ef4-497e-b4aa-e06b3db670f5" (UID: "6a1bbc81-7ef4-497e-b4aa-e06b3db670f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:50:58.022322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.022286 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-plugins-config-volume" pod "64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c" (UID: "64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:50:58.022716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.022697 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:58.022716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.022716 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:58.024299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.024276 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-kube-api-access-jwsnm" pod "6a1bbc81-7ef4-497e-b4aa-e06b3db670f5" (UID: "6a1bbc81-7ef4-497e-b4aa-e06b3db670f5"). InnerVolumeSpecName "kube-api-access-jwsnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:50:58.024299 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.024289 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-kube-api-access-66tht" pod "64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c" (UID: "64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c"). InnerVolumeSpecName "kube-api-access-66tht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:50:58.123871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.123843 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66tht\" (UniqueName: \"kubernetes.io/projected/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c-kube-api-access-66tht\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:58.123871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.123869 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwsnm\" (UniqueName: \"kubernetes.io/projected/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5-kube-api-access-jwsnm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:58.523617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.523564 5532 generic.go:356] "Generic (PLEG): container finished" podID="64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c" containerID="641c80cb5498a5d776bdcf2047ea39fb9f542950256064964f6a451047a139ed" exitCode=0 Oct 31 13:50:58.524105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.523618 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" Oct 31 13:50:58.524105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.523649 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" event={"ID":"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c","Type":"ContainerDied","Data":"641c80cb5498a5d776bdcf2047ea39fb9f542950256064964f6a451047a139ed"} Oct 31 13:50:58.524105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.523707 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44" event={"ID":"64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c","Type":"ContainerDied","Data":"6e27ad8d6d15fefc848c7bccb86236f7dc70217155a5fe58cb94926f83e2c8fb"} Oct 31 13:50:58.524105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.523735 5532 scope.go:122] "RemoveContainer" containerID="641c80cb5498a5d776bdcf2047ea39fb9f542950256064964f6a451047a139ed" Oct 31 13:50:58.525536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.525507 5532 generic.go:356] "Generic (PLEG): container finished" podID="6a1bbc81-7ef4-497e-b4aa-e06b3db670f5" containerID="ddeeceeac7253ba5ab475834d9721a9cc3b33ffbecf9788b7ad2b42fb21bcfb8" exitCode=0 Oct 31 13:50:58.525731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.525567 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" Oct 31 13:50:58.525731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.525635 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" event={"ID":"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5","Type":"ContainerDied","Data":"ddeeceeac7253ba5ab475834d9721a9cc3b33ffbecf9788b7ad2b42fb21bcfb8"} Oct 31 13:50:58.525731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.525686 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8" event={"ID":"6a1bbc81-7ef4-497e-b4aa-e06b3db670f5","Type":"ContainerDied","Data":"4e78b901de1e6c6bfb17c124d24ba693a8b69f198cb8932506c0127f3e5a5452"} Oct 31 13:50:58.529401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.529375 5532 generic.go:356] "Generic (PLEG): container finished" podID="832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2" containerID="04f26a39638f311cb8299e54e5c0aff68361a84febbba34899f58303f0b96554" exitCode=0 Oct 31 13:50:58.529521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.529424 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" event={"ID":"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2","Type":"ContainerDied","Data":"04f26a39638f311cb8299e54e5c0aff68361a84febbba34899f58303f0b96554"} Oct 31 13:50:58.539251 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.539223 5532 scope.go:122] "RemoveContainer" containerID="641c80cb5498a5d776bdcf2047ea39fb9f542950256064964f6a451047a139ed" Oct 31 13:50:58.539678 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:50:58.539647 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"641c80cb5498a5d776bdcf2047ea39fb9f542950256064964f6a451047a139ed\": container with ID starting with 641c80cb5498a5d776bdcf2047ea39fb9f542950256064964f6a451047a139ed not found: ID does not exist" containerID="641c80cb5498a5d776bdcf2047ea39fb9f542950256064964f6a451047a139ed" Oct 31 13:50:58.539780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.539689 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"641c80cb5498a5d776bdcf2047ea39fb9f542950256064964f6a451047a139ed"} err="failed to get container status \"641c80cb5498a5d776bdcf2047ea39fb9f542950256064964f6a451047a139ed\": rpc error: code = NotFound desc = could not find container \"641c80cb5498a5d776bdcf2047ea39fb9f542950256064964f6a451047a139ed\": container with ID starting with 641c80cb5498a5d776bdcf2047ea39fb9f542950256064964f6a451047a139ed not found: ID does not exist" Oct 31 13:50:58.539780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.539715 5532 scope.go:122] "RemoveContainer" containerID="ddeeceeac7253ba5ab475834d9721a9cc3b33ffbecf9788b7ad2b42fb21bcfb8" Oct 31 13:50:58.554992 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.554969 5532 scope.go:122] "RemoveContainer" containerID="ddeeceeac7253ba5ab475834d9721a9cc3b33ffbecf9788b7ad2b42fb21bcfb8" Oct 31 13:50:58.555342 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:50:58.555321 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddeeceeac7253ba5ab475834d9721a9cc3b33ffbecf9788b7ad2b42fb21bcfb8\": container with ID starting with ddeeceeac7253ba5ab475834d9721a9cc3b33ffbecf9788b7ad2b42fb21bcfb8 not found: ID does not exist" containerID="ddeeceeac7253ba5ab475834d9721a9cc3b33ffbecf9788b7ad2b42fb21bcfb8" Oct 31 13:50:58.555430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.555353 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddeeceeac7253ba5ab475834d9721a9cc3b33ffbecf9788b7ad2b42fb21bcfb8"} err="failed to get container status \"ddeeceeac7253ba5ab475834d9721a9cc3b33ffbecf9788b7ad2b42fb21bcfb8\": rpc error: code = NotFound desc = could not find container \"ddeeceeac7253ba5ab475834d9721a9cc3b33ffbecf9788b7ad2b42fb21bcfb8\": container with ID starting with ddeeceeac7253ba5ab475834d9721a9cc3b33ffbecf9788b7ad2b42fb21bcfb8 not found: ID does not exist" Oct 31 13:50:58.584028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.583996 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44"] Oct 31 13:50:58.589194 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.589168 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-35b584-655c54f7ff-tqq44"] Oct 31 13:50:58.602024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.601995 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8"] Oct 31 13:50:58.609509 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:58.609477 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-35b584-5b757cf5d5-ntgf8"] Oct 31 13:50:59.714109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:59.714085 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" Oct 31 13:50:59.747409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:59.747379 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-kube-api-access-rsrgt\" (UniqueName: \"kubernetes.io/projected/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-kube-api-access-rsrgt\") pod \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\" (UID: \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\") " Oct 31 13:50:59.747604 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:59.747574 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-openshift-ca\") pod \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\" (UID: \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\") " Oct 31 13:50:59.747676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:59.747654 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-xks-ca\") pod \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\" (UID: \"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2\") " Oct 31 13:50:59.748011 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:59.747978 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-openshift-ca" pod "832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2" (UID: "832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:50:59.748357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:59.748337 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:59.749833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:59.749804 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-kube-api-access-rsrgt" pod "832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2" (UID: "832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2"). InnerVolumeSpecName "kube-api-access-rsrgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:50:59.750046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:59.750023 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-xks-ca" pod "832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2" (UID: "832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:50:59.849809 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:59.849733 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:50:59.849809 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:50:59.849761 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsrgt\" (UniqueName: \"kubernetes.io/projected/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2-kube-api-access-rsrgt\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:00.005768 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:00.005733 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c" path="/var/lib/kubelet/pods/64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c/volumes" Oct 31 13:51:00.006295 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:00.006278 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a1bbc81-7ef4-497e-b4aa-e06b3db670f5" path="/var/lib/kubelet/pods/6a1bbc81-7ef4-497e-b4aa-e06b3db670f5/volumes" Oct 31 13:51:00.550999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:00.550971 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" Oct 31 13:51:00.551155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:00.551001 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6" event={"ID":"832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2","Type":"ContainerDied","Data":"73f341c5e976045b92c140431e9c391384b5baa5dcb13ffc8c84b3012cdd4ffa"} Oct 31 13:51:00.551155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:00.551050 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73f341c5e976045b92c140431e9c391384b5baa5dcb13ffc8c84b3012cdd4ffa" Oct 31 13:51:01.762785 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:51:01.762745 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:51:14.062761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.062726 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw"] Oct 31 13:51:14.063589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.063567 5532 cpu_manager.go:414] "containerMap: removing container" podUID="832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2" containerName="revoke-keys" Oct 31 13:51:14.063589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.063616 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2" containerName="revoke-keys" Oct 31 13:51:14.063759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.063678 5532 cpu_manager.go:414] "containerMap: removing container" podUID="64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c" containerName="payload-pre-processing" Oct 31 13:51:14.063759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.063688 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c" containerName="payload-pre-processing" Oct 31 13:51:14.063759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.063711 5532 cpu_manager.go:414] "containerMap: removing container" podUID="6a1bbc81-7ef4-497e-b4aa-e06b3db670f5" containerName="payload-processing" Oct 31 13:51:14.063759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.063718 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a1bbc81-7ef4-497e-b4aa-e06b3db670f5" containerName="payload-processing" Oct 31 13:51:14.063939 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.063924 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="64e27b8e-58f0-41e7-a8e4-eb7c403b5b2c" containerName="payload-pre-processing" Oct 31 13:51:14.063979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.063943 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a1bbc81-7ef4-497e-b4aa-e06b3db670f5" containerName="payload-processing" Oct 31 13:51:14.063979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.063960 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2" containerName="revoke-keys" Oct 31 13:51:14.069141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.069123 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" Oct 31 13:51:14.076073 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.076046 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:51:14.076192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.076137 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:51:14.081223 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.081196 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw"] Oct 31 13:51:14.132491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.132459 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8622936-7b7kw\" (UID: \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" Oct 31 13:51:14.132692 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.132514 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxg9q\" (UniqueName: \"kubernetes.io/projected/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-kube-api-access-sxg9q\") pod \"maas-api-revoke-sub-d8622936-7b7kw\" (UID: \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" Oct 31 13:51:14.132767 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.132747 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8622936-7b7kw\" (UID: \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" Oct 31 13:51:14.234745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.234700 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8622936-7b7kw\" (UID: \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" Oct 31 13:51:14.234976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.234957 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8622936-7b7kw\" (UID: \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" Oct 31 13:51:14.235055 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.235010 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxg9q\" (UniqueName: \"kubernetes.io/projected/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-kube-api-access-sxg9q\") pod \"maas-api-revoke-sub-d8622936-7b7kw\" (UID: \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" Oct 31 13:51:14.235408 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.235385 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8622936-7b7kw\" (UID: \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" Oct 31 13:51:14.237632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.237574 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8622936-7b7kw\" (UID: \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" Oct 31 13:51:14.260401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.260375 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxg9q\" (UniqueName: \"kubernetes.io/projected/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-kube-api-access-sxg9q\") pod \"maas-api-revoke-sub-d8622936-7b7kw\" (UID: \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" Oct 31 13:51:14.386515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.386421 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" Oct 31 13:51:14.684913 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:51:14.684882 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb726aed_e9aa_4bb2_98ac_3f3ddc8686a6.slice/crio-8ffa270863c823d0839d85e546e89b9d43227d3b4f0b1931163497dd70ad7f57 WatchSource:0}: Error finding container 8ffa270863c823d0839d85e546e89b9d43227d3b4f0b1931163497dd70ad7f57: Status 404 returned error can't find the container with id 8ffa270863c823d0839d85e546e89b9d43227d3b4f0b1931163497dd70ad7f57 Oct 31 13:51:14.685062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:14.684964 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw"] Oct 31 13:51:15.705103 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:15.705069 5532 generic.go:356] "Generic (PLEG): container finished" podID="eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6" containerID="fe949cdf8b9959ad1204d1a784fb95417a95259fe741c9025b786b97d8ee965c" exitCode=0 Oct 31 13:51:15.705517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:15.705127 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" event={"ID":"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6","Type":"ContainerDied","Data":"fe949cdf8b9959ad1204d1a784fb95417a95259fe741c9025b786b97d8ee965c"} Oct 31 13:51:15.705517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:15.705169 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" event={"ID":"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6","Type":"ContainerStarted","Data":"8ffa270863c823d0839d85e546e89b9d43227d3b4f0b1931163497dd70ad7f57"} Oct 31 13:51:16.887159 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:16.887134 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" Oct 31 13:51:17.014217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:17.014117 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-kube-api-access-sxg9q\" (UniqueName: \"kubernetes.io/projected/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-kube-api-access-sxg9q\") pod \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\" (UID: \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\") " Oct 31 13:51:17.014217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:17.014165 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-xks-ca\") pod \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\" (UID: \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\") " Oct 31 13:51:17.014513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:17.014287 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-openshift-ca\") pod \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\" (UID: \"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6\") " Oct 31 13:51:17.014703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:17.014669 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-openshift-ca" pod "eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6" (UID: "eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:51:17.015179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:17.015157 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:17.016524 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:17.016501 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-xks-ca" pod "eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6" (UID: "eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:51:17.016634 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:17.016557 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-kube-api-access-sxg9q" pod "eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6" (UID: "eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6"). InnerVolumeSpecName "kube-api-access-sxg9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:51:17.117307 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:17.117273 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:17.117307 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:17.117307 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxg9q\" (UniqueName: \"kubernetes.io/projected/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6-kube-api-access-sxg9q\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:17.724260 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:17.724222 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" Oct 31 13:51:17.724260 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:17.724244 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw" event={"ID":"eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6","Type":"ContainerDied","Data":"8ffa270863c823d0839d85e546e89b9d43227d3b4f0b1931163497dd70ad7f57"} Oct 31 13:51:17.724483 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:17.724302 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ffa270863c823d0839d85e546e89b9d43227d3b4f0b1931163497dd70ad7f57" Oct 31 13:51:20.650218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:20.650183 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:51:31.792845 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:51:31.792802 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:51:31.795673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:31.795642 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp"] Oct 31 13:51:31.796870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:31.796847 5532 cpu_manager.go:414] "containerMap: removing container" podUID="eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6" containerName="revoke-keys" Oct 31 13:51:31.796870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:31.796870 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6" containerName="revoke-keys" Oct 31 13:51:31.797105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:31.797092 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6" containerName="revoke-keys" Oct 31 13:51:31.810797 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:31.810754 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp"] Oct 31 13:51:31.810985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:31.810875 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" Oct 31 13:51:31.824073 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:31.824043 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:51:31.855650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:31.826293 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:51:31.902754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:31.902723 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp\" (UID: \"62108e88-1f7a-4a04-b4ed-8177b9af6237\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" Oct 31 13:51:31.902930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:31.902795 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp\" (UID: \"62108e88-1f7a-4a04-b4ed-8177b9af6237\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" Oct 31 13:51:31.902975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:31.902923 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpk9k\" (UniqueName: \"kubernetes.io/projected/62108e88-1f7a-4a04-b4ed-8177b9af6237-kube-api-access-kpk9k\") pod \"maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp\" (UID: \"62108e88-1f7a-4a04-b4ed-8177b9af6237\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" Oct 31 13:51:32.004303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:32.004276 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp\" (UID: \"62108e88-1f7a-4a04-b4ed-8177b9af6237\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" Oct 31 13:51:32.004493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:32.004342 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp\" (UID: \"62108e88-1f7a-4a04-b4ed-8177b9af6237\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" Oct 31 13:51:32.004493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:32.004400 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpk9k\" (UniqueName: \"kubernetes.io/projected/62108e88-1f7a-4a04-b4ed-8177b9af6237-kube-api-access-kpk9k\") pod \"maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp\" (UID: \"62108e88-1f7a-4a04-b4ed-8177b9af6237\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" Oct 31 13:51:32.005120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:32.005086 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp\" (UID: \"62108e88-1f7a-4a04-b4ed-8177b9af6237\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" Oct 31 13:51:32.006894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:32.006863 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp\" (UID: \"62108e88-1f7a-4a04-b4ed-8177b9af6237\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" Oct 31 13:51:32.021436 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:32.021405 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpk9k\" (UniqueName: \"kubernetes.io/projected/62108e88-1f7a-4a04-b4ed-8177b9af6237-kube-api-access-kpk9k\") pod \"maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp\" (UID: \"62108e88-1f7a-4a04-b4ed-8177b9af6237\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" Oct 31 13:51:32.167207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:32.167169 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" Oct 31 13:51:32.461820 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:51:32.461789 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62108e88_1f7a_4a04_b4ed_8177b9af6237.slice/crio-74b0332c2e58d3af2bcb2e9f7858808327d66616d555743217dd6aef627036a2 WatchSource:0}: Error finding container 74b0332c2e58d3af2bcb2e9f7858808327d66616d555743217dd6aef627036a2: Status 404 returned error can't find the container with id 74b0332c2e58d3af2bcb2e9f7858808327d66616d555743217dd6aef627036a2 Oct 31 13:51:32.462010 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:32.461985 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp"] Oct 31 13:51:32.884250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:32.884210 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" event={"ID":"62108e88-1f7a-4a04-b4ed-8177b9af6237","Type":"ContainerStarted","Data":"e2ca391b9de05f49d333874beb26c0d10ff7279d52d9073b1b3f45489d9aa133"} Oct 31 13:51:32.884584 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:32.884279 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" event={"ID":"62108e88-1f7a-4a04-b4ed-8177b9af6237","Type":"ContainerStarted","Data":"74b0332c2e58d3af2bcb2e9f7858808327d66616d555743217dd6aef627036a2"} Oct 31 13:51:32.907365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:32.907295 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" podStartSLOduration=1.907282322 podStartE2EDuration="1.907282322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:51:32.903775824 +0000 UTC m=+2861.049519652" watchObservedRunningTime="2026-10-31 13:51:32.907282322 +0000 UTC m=+2861.053026160" Oct 31 13:51:33.894282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:33.894244 5532 generic.go:356] "Generic (PLEG): container finished" podID="62108e88-1f7a-4a04-b4ed-8177b9af6237" containerID="e2ca391b9de05f49d333874beb26c0d10ff7279d52d9073b1b3f45489d9aa133" exitCode=0 Oct 31 13:51:33.894715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:33.894315 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" event={"ID":"62108e88-1f7a-4a04-b4ed-8177b9af6237","Type":"ContainerDied","Data":"e2ca391b9de05f49d333874beb26c0d10ff7279d52d9073b1b3f45489d9aa133"} Oct 31 13:51:35.077411 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:35.077376 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" Oct 31 13:51:35.157884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:35.157788 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-openshift-ca\") pod \"62108e88-1f7a-4a04-b4ed-8177b9af6237\" (UID: \"62108e88-1f7a-4a04-b4ed-8177b9af6237\") " Oct 31 13:51:35.158072 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:35.157929 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-xks-ca\") pod \"62108e88-1f7a-4a04-b4ed-8177b9af6237\" (UID: \"62108e88-1f7a-4a04-b4ed-8177b9af6237\") " Oct 31 13:51:35.158072 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:35.158031 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62108e88-1f7a-4a04-b4ed-8177b9af6237-kube-api-access-kpk9k\" (UniqueName: \"kubernetes.io/projected/62108e88-1f7a-4a04-b4ed-8177b9af6237-kube-api-access-kpk9k\") pod \"62108e88-1f7a-4a04-b4ed-8177b9af6237\" (UID: \"62108e88-1f7a-4a04-b4ed-8177b9af6237\") " Oct 31 13:51:35.158220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:35.158197 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-openshift-ca" pod "62108e88-1f7a-4a04-b4ed-8177b9af6237" (UID: "62108e88-1f7a-4a04-b4ed-8177b9af6237"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:51:35.158772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:35.158717 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:35.160094 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:35.160073 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-xks-ca" pod "62108e88-1f7a-4a04-b4ed-8177b9af6237" (UID: "62108e88-1f7a-4a04-b4ed-8177b9af6237"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:51:35.160236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:35.160222 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62108e88-1f7a-4a04-b4ed-8177b9af6237-kube-api-access-kpk9k" pod "62108e88-1f7a-4a04-b4ed-8177b9af6237" (UID: "62108e88-1f7a-4a04-b4ed-8177b9af6237"). InnerVolumeSpecName "kube-api-access-kpk9k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:51:35.260290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:35.260253 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpk9k\" (UniqueName: \"kubernetes.io/projected/62108e88-1f7a-4a04-b4ed-8177b9af6237-kube-api-access-kpk9k\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:35.260290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:35.260287 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62108e88-1f7a-4a04-b4ed-8177b9af6237-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:51:35.911488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:35.911452 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" event={"ID":"62108e88-1f7a-4a04-b4ed-8177b9af6237","Type":"ContainerDied","Data":"74b0332c2e58d3af2bcb2e9f7858808327d66616d555743217dd6aef627036a2"} Oct 31 13:51:35.911488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:35.911490 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp" Oct 31 13:51:35.911740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:35.911505 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74b0332c2e58d3af2bcb2e9f7858808327d66616d555743217dd6aef627036a2" Oct 31 13:51:41.930588 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:41.930547 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht"] Oct 31 13:51:46.931465 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:46.931406 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" podUID="30d3f968-575d-4bb8-8e8b-8918e9f03ed5" containerName="payload-processing" containerID="cri-o://8218824cc36e51b2dbf919fecff6a9903dd70aa2f944acb5fec2873bab9feee8" gracePeriod=25 Oct 31 13:51:47.014332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:47.014295 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" podUID="30d3f968-575d-4bb8-8e8b-8918e9f03ed5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:51:54.971542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:54.971511 5532 scope.go:122] "RemoveContainer" containerID="b1173a54f1cd3a04c4c7476393a38424d98bfde31edc4e4cba0e9bd4e114f256" Oct 31 13:51:54.986216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:54.986184 5532 scope.go:122] "RemoveContainer" containerID="89946794e19a6ee0a3fd63d6a029cc3990f66545896b749c51f1924b8d399888" Oct 31 13:51:54.997117 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:54.997097 5532 scope.go:122] "RemoveContainer" containerID="3445af2d9cff82b521d3fafaa79bcb56e3f67233c1b7f756e497858252fdd6a2" Oct 31 13:51:55.007178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:55.007157 5532 scope.go:122] "RemoveContainer" containerID="041de7b4d31e558b71225af9e41ab82253c1afb87bd36d8314063aaee1c65c4b" Oct 31 13:51:57.014406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:51:57.014362 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" podUID="30d3f968-575d-4bb8-8e8b-8918e9f03ed5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:52:00.036290 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:00.036206 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp"] Oct 31 13:52:00.040067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:00.040026 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1ddade-rhrtp"] Oct 31 13:52:01.764018 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:52:01.763981 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:52:02.009694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:02.009657 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62108e88-1f7a-4a04-b4ed-8177b9af6237" path="/var/lib/kubelet/pods/62108e88-1f7a-4a04-b4ed-8177b9af6237/volumes" Oct 31 13:52:07.014490 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:52:07.014458 5532 logging.go:55] [core] [Channel #8135 SubChannel #8136]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.173:50052", ServerName: "10.132.0.173:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.173:50052: connect: connection refused" Oct 31 13:52:07.197086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:07.197059 5532 generic.go:356] "Generic (PLEG): container finished" podID="30d3f968-575d-4bb8-8e8b-8918e9f03ed5" containerID="8218824cc36e51b2dbf919fecff6a9903dd70aa2f944acb5fec2873bab9feee8" exitCode=0 Oct 31 13:52:07.197211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:07.197154 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" event={"ID":"30d3f968-575d-4bb8-8e8b-8918e9f03ed5","Type":"ContainerDied","Data":"8218824cc36e51b2dbf919fecff6a9903dd70aa2f944acb5fec2873bab9feee8"} Oct 31 13:52:07.197281 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:07.197256 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" event={"ID":"30d3f968-575d-4bb8-8e8b-8918e9f03ed5","Type":"ContainerDied","Data":"b3dfda3253664cd41498c909225f2eca63ee161ca1d6729415a19df7095e7c80"} Oct 31 13:52:07.197328 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:07.197283 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3dfda3253664cd41498c909225f2eca63ee161ca1d6729415a19df7095e7c80" Oct 31 13:52:07.220038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:07.220016 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" Oct 31 13:52:07.350180 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:07.350145 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-plugins-config-volume\") pod \"30d3f968-575d-4bb8-8e8b-8918e9f03ed5\" (UID: \"30d3f968-575d-4bb8-8e8b-8918e9f03ed5\") " Oct 31 13:52:07.350371 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:07.350249 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-kube-api-access-dsm9d\" (UniqueName: \"kubernetes.io/projected/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-kube-api-access-dsm9d\") pod \"30d3f968-575d-4bb8-8e8b-8918e9f03ed5\" (UID: \"30d3f968-575d-4bb8-8e8b-8918e9f03ed5\") " Oct 31 13:52:07.350556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:07.350528 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-plugins-config-volume" pod "30d3f968-575d-4bb8-8e8b-8918e9f03ed5" (UID: "30d3f968-575d-4bb8-8e8b-8918e9f03ed5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:52:07.350954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:07.350936 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:07.352403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:07.352384 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-kube-api-access-dsm9d" pod "30d3f968-575d-4bb8-8e8b-8918e9f03ed5" (UID: "30d3f968-575d-4bb8-8e8b-8918e9f03ed5"). InnerVolumeSpecName "kube-api-access-dsm9d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:52:07.453649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:07.453438 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsm9d\" (UniqueName: \"kubernetes.io/projected/30d3f968-575d-4bb8-8e8b-8918e9f03ed5-kube-api-access-dsm9d\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:08.015020 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.014989 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" podUID="30d3f968-575d-4bb8-8e8b-8918e9f03ed5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.173:50052\" within 1s: context deadline exceeded" Oct 31 13:52:08.015332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.015077 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" Oct 31 13:52:08.173401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.173370 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj"] Oct 31 13:52:08.174169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.174154 5532 cpu_manager.go:414] "containerMap: removing container" podUID="30d3f968-575d-4bb8-8e8b-8918e9f03ed5" containerName="payload-processing" Oct 31 13:52:08.174227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.174170 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30d3f968-575d-4bb8-8e8b-8918e9f03ed5" containerName="payload-processing" Oct 31 13:52:08.174227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.174185 5532 cpu_manager.go:414] "containerMap: removing container" podUID="62108e88-1f7a-4a04-b4ed-8177b9af6237" containerName="revoke-keys" Oct 31 13:52:08.174227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.174191 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62108e88-1f7a-4a04-b4ed-8177b9af6237" containerName="revoke-keys" Oct 31 13:52:08.174351 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.174345 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="30d3f968-575d-4bb8-8e8b-8918e9f03ed5" containerName="payload-processing" Oct 31 13:52:08.174388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.174355 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="62108e88-1f7a-4a04-b4ed-8177b9af6237" containerName="revoke-keys" Oct 31 13:52:08.178262 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.178243 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" Oct 31 13:52:08.183882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.183859 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:52:08.184051 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.183989 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:52:08.188174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.188149 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj"] Oct 31 13:52:08.206399 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.206368 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht" Oct 31 13:52:08.231227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.231196 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht"] Oct 31 13:52:08.239420 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.239383 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1ddade-7c4f444ff9-plwht"] Oct 31 13:52:08.301968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.301876 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72c210-slzjj\" (UID: \"f548adc7-0a93-47df-b5b6-d2084ddc3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" Oct 31 13:52:08.301968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.301931 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxzrm\" (UniqueName: \"kubernetes.io/projected/f548adc7-0a93-47df-b5b6-d2084ddc3111-kube-api-access-wxzrm\") pod \"maas-api-revoke-keys-e2e-inf-b-72c210-slzjj\" (UID: \"f548adc7-0a93-47df-b5b6-d2084ddc3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" Oct 31 13:52:08.302279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.302248 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72c210-slzjj\" (UID: \"f548adc7-0a93-47df-b5b6-d2084ddc3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" Oct 31 13:52:08.403356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.403321 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72c210-slzjj\" (UID: \"f548adc7-0a93-47df-b5b6-d2084ddc3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" Oct 31 13:52:08.403550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.403379 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72c210-slzjj\" (UID: \"f548adc7-0a93-47df-b5b6-d2084ddc3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" Oct 31 13:52:08.403550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.403430 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxzrm\" (UniqueName: \"kubernetes.io/projected/f548adc7-0a93-47df-b5b6-d2084ddc3111-kube-api-access-wxzrm\") pod \"maas-api-revoke-keys-e2e-inf-b-72c210-slzjj\" (UID: \"f548adc7-0a93-47df-b5b6-d2084ddc3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" Oct 31 13:52:08.404040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.404014 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72c210-slzjj\" (UID: \"f548adc7-0a93-47df-b5b6-d2084ddc3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" Oct 31 13:52:08.405940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.405917 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-72c210-slzjj\" (UID: \"f548adc7-0a93-47df-b5b6-d2084ddc3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" Oct 31 13:52:08.417240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.417213 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxzrm\" (UniqueName: \"kubernetes.io/projected/f548adc7-0a93-47df-b5b6-d2084ddc3111-kube-api-access-wxzrm\") pod \"maas-api-revoke-keys-e2e-inf-b-72c210-slzjj\" (UID: \"f548adc7-0a93-47df-b5b6-d2084ddc3111\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" Oct 31 13:52:08.495421 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.495386 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" Oct 31 13:52:08.797811 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:52:08.797766 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf548adc7_0a93_47df_b5b6_d2084ddc3111.slice/crio-2ddc344c554b37afc253fa160bba2946ac8acd170eb540675dcb4961f2fb5b1d WatchSource:0}: Error finding container 2ddc344c554b37afc253fa160bba2946ac8acd170eb540675dcb4961f2fb5b1d: Status 404 returned error can't find the container with id 2ddc344c554b37afc253fa160bba2946ac8acd170eb540675dcb4961f2fb5b1d Oct 31 13:52:08.797938 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:08.797831 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj"] Oct 31 13:52:09.215696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:09.215649 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" event={"ID":"f548adc7-0a93-47df-b5b6-d2084ddc3111","Type":"ContainerStarted","Data":"2ddc344c554b37afc253fa160bba2946ac8acd170eb540675dcb4961f2fb5b1d"} Oct 31 13:52:10.005483 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:10.005448 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30d3f968-575d-4bb8-8e8b-8918e9f03ed5" path="/var/lib/kubelet/pods/30d3f968-575d-4bb8-8e8b-8918e9f03ed5/volumes" Oct 31 13:52:10.225554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:10.225470 5532 generic.go:356] "Generic (PLEG): container finished" podID="f548adc7-0a93-47df-b5b6-d2084ddc3111" containerID="b9844fd223b204679d886bab6fb58ea4451ee6a5caadb63bb609ea6af8cbd69b" exitCode=0 Oct 31 13:52:10.225957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:10.225552 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" event={"ID":"f548adc7-0a93-47df-b5b6-d2084ddc3111","Type":"ContainerDied","Data":"b9844fd223b204679d886bab6fb58ea4451ee6a5caadb63bb609ea6af8cbd69b"} Oct 31 13:52:11.417120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:11.417090 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" Oct 31 13:52:11.475199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:11.475164 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-openshift-ca\") pod \"f548adc7-0a93-47df-b5b6-d2084ddc3111\" (UID: \"f548adc7-0a93-47df-b5b6-d2084ddc3111\") " Oct 31 13:52:11.475404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:11.475275 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f548adc7-0a93-47df-b5b6-d2084ddc3111-kube-api-access-wxzrm\" (UniqueName: \"kubernetes.io/projected/f548adc7-0a93-47df-b5b6-d2084ddc3111-kube-api-access-wxzrm\") pod \"f548adc7-0a93-47df-b5b6-d2084ddc3111\" (UID: \"f548adc7-0a93-47df-b5b6-d2084ddc3111\") " Oct 31 13:52:11.475404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:11.475328 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-xks-ca\") pod \"f548adc7-0a93-47df-b5b6-d2084ddc3111\" (UID: \"f548adc7-0a93-47df-b5b6-d2084ddc3111\") " Oct 31 13:52:11.475704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:11.475589 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-openshift-ca" pod "f548adc7-0a93-47df-b5b6-d2084ddc3111" (UID: "f548adc7-0a93-47df-b5b6-d2084ddc3111"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:52:11.476217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:11.476195 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:11.478072 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:11.478042 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-xks-ca" pod "f548adc7-0a93-47df-b5b6-d2084ddc3111" (UID: "f548adc7-0a93-47df-b5b6-d2084ddc3111"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:52:11.478194 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:11.478073 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f548adc7-0a93-47df-b5b6-d2084ddc3111-kube-api-access-wxzrm" pod "f548adc7-0a93-47df-b5b6-d2084ddc3111" (UID: "f548adc7-0a93-47df-b5b6-d2084ddc3111"). InnerVolumeSpecName "kube-api-access-wxzrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:52:11.577649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:11.577616 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxzrm\" (UniqueName: \"kubernetes.io/projected/f548adc7-0a93-47df-b5b6-d2084ddc3111-kube-api-access-wxzrm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:11.577649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:11.577645 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f548adc7-0a93-47df-b5b6-d2084ddc3111-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:12.245332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:12.245295 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" event={"ID":"f548adc7-0a93-47df-b5b6-d2084ddc3111","Type":"ContainerDied","Data":"2ddc344c554b37afc253fa160bba2946ac8acd170eb540675dcb4961f2fb5b1d"} Oct 31 13:52:12.245506 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:12.245353 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ddc344c554b37afc253fa160bba2946ac8acd170eb540675dcb4961f2fb5b1d" Oct 31 13:52:12.245506 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:12.245314 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj" Oct 31 13:52:25.443228 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:25.443195 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj"] Oct 31 13:52:25.449783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:25.449757 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-72c210-slzjj"] Oct 31 13:52:26.004726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:26.004696 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f548adc7-0a93-47df-b5b6-d2084ddc3111" path="/var/lib/kubelet/pods/f548adc7-0a93-47df-b5b6-d2084ddc3111/volumes" Oct 31 13:52:27.489067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:27.489035 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz"] Oct 31 13:52:27.489515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:27.489254 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" podUID="75407a66-4806-493d-a535-bfaaaddfbe16" containerName="istio-proxy" containerID="cri-o://730bc018d3138d424b7ee1d4dffdf25b9a8c4c03fe21a761932e721e1bc73394" gracePeriod=30 Oct 31 13:52:31.763075 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:52:31.763035 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:52:31.893463 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:31.893428 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85"] Oct 31 13:52:31.893710 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:31.893653 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerName="istio-proxy" containerID="cri-o://9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa" gracePeriod=30 Oct 31 13:52:32.782413 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.782381 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:52:32.886366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.886334 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-envoy\") pod \"75407a66-4806-493d-a535-bfaaaddfbe16\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " Oct 31 13:52:32.886571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.886407 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-socket\") pod \"75407a66-4806-493d-a535-bfaaaddfbe16\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " Oct 31 13:52:32.886571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.886431 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-kube-api-access-gs2n8\" (UniqueName: \"kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-kube-api-access-gs2n8\") pod \"75407a66-4806-493d-a535-bfaaaddfbe16\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " Oct 31 13:52:32.886571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.886466 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75407a66-4806-493d-a535-bfaaaddfbe16-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75407a66-4806-493d-a535-bfaaaddfbe16-istiod-ca-cert\") pod \"75407a66-4806-493d-a535-bfaaaddfbe16\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " Oct 31 13:52:32.886571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.886507 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-credential-socket\") pod \"75407a66-4806-493d-a535-bfaaaddfbe16\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " Oct 31 13:52:32.886874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.886635 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-data\") pod \"75407a66-4806-493d-a535-bfaaaddfbe16\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " Oct 31 13:52:32.886874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.886719 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-certs\") pod \"75407a66-4806-493d-a535-bfaaaddfbe16\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " Oct 31 13:52:32.886874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.886806 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-socket" pod "75407a66-4806-493d-a535-bfaaaddfbe16" (UID: "75407a66-4806-493d-a535-bfaaaddfbe16"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:32.886874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.886825 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-istio-token\" (UniqueName: \"kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-istio-token\") pod \"75407a66-4806-493d-a535-bfaaaddfbe16\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " Oct 31 13:52:32.887111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.886942 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75407a66-4806-493d-a535-bfaaaddfbe16-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75407a66-4806-493d-a535-bfaaaddfbe16-istio-podinfo\") pod \"75407a66-4806-493d-a535-bfaaaddfbe16\" (UID: \"75407a66-4806-493d-a535-bfaaaddfbe16\") " Oct 31 13:52:32.887111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.886941 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-credential-socket" pod "75407a66-4806-493d-a535-bfaaaddfbe16" (UID: "75407a66-4806-493d-a535-bfaaaddfbe16"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:32.887111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.886959 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75407a66-4806-493d-a535-bfaaaddfbe16-istiod-ca-cert" pod "75407a66-4806-493d-a535-bfaaaddfbe16" (UID: "75407a66-4806-493d-a535-bfaaaddfbe16"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:52:32.887111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.887077 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-certs" pod "75407a66-4806-493d-a535-bfaaaddfbe16" (UID: "75407a66-4806-493d-a535-bfaaaddfbe16"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:32.887347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.887204 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-data" pod "75407a66-4806-493d-a535-bfaaaddfbe16" (UID: "75407a66-4806-493d-a535-bfaaaddfbe16"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:32.887645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.887625 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:32.887645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.887646 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:32.888012 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.887655 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75407a66-4806-493d-a535-bfaaaddfbe16-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:32.888012 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.887664 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:32.888012 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.887672 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:32.888738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.888712 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-envoy" pod "75407a66-4806-493d-a535-bfaaaddfbe16" (UID: "75407a66-4806-493d-a535-bfaaaddfbe16"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:32.889112 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.889094 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-kube-api-access-gs2n8" pod "75407a66-4806-493d-a535-bfaaaddfbe16" (UID: "75407a66-4806-493d-a535-bfaaaddfbe16"). InnerVolumeSpecName "kube-api-access-gs2n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:52:32.889398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.889376 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-istio-token" pod "75407a66-4806-493d-a535-bfaaaddfbe16" (UID: "75407a66-4806-493d-a535-bfaaaddfbe16"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:52:32.889398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.889377 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75407a66-4806-493d-a535-bfaaaddfbe16-istio-podinfo" pod "75407a66-4806-493d-a535-bfaaaddfbe16" (UID: "75407a66-4806-493d-a535-bfaaaddfbe16"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:52:32.989485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.989451 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:32.989485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.989481 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75407a66-4806-493d-a535-bfaaaddfbe16-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:32.989485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.989491 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75407a66-4806-493d-a535-bfaaaddfbe16-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:32.989760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:32.989500 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs2n8\" (UniqueName: \"kubernetes.io/projected/75407a66-4806-493d-a535-bfaaaddfbe16-kube-api-access-gs2n8\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:33.446009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:33.445977 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz"] Oct 31 13:52:33.446259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:33.446222 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" containerID="cri-o://aa7d56d5bddbe82f00fbb10b4cc4609b710e4d4835da81b8b6bb7877be858444" gracePeriod=30 Oct 31 13:52:33.446259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:33.446255 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" event={"ID":"75407a66-4806-493d-a535-bfaaaddfbe16","Type":"ContainerDied","Data":"730bc018d3138d424b7ee1d4dffdf25b9a8c4c03fe21a761932e721e1bc73394"} Oct 31 13:52:33.446484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:33.446309 5532 scope.go:122] "RemoveContainer" containerID="730bc018d3138d424b7ee1d4dffdf25b9a8c4c03fe21a761932e721e1bc73394" Oct 31 13:52:33.446484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:33.446238 5532 generic.go:356] "Generic (PLEG): container finished" podID="75407a66-4806-493d-a535-bfaaaddfbe16" containerID="730bc018d3138d424b7ee1d4dffdf25b9a8c4c03fe21a761932e721e1bc73394" exitCode=0 Oct 31 13:52:33.446484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:33.446411 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" Oct 31 13:52:33.446719 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:33.446410 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz" event={"ID":"75407a66-4806-493d-a535-bfaaaddfbe16","Type":"ContainerDied","Data":"5c0a3b361c22aa128191aa3d36b89769d73ad3546ad0028d2d13a6e100aa32d9"} Oct 31 13:52:33.458978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:33.458960 5532 scope.go:122] "RemoveContainer" containerID="730bc018d3138d424b7ee1d4dffdf25b9a8c4c03fe21a761932e721e1bc73394" Oct 31 13:52:33.459253 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:52:33.459232 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730bc018d3138d424b7ee1d4dffdf25b9a8c4c03fe21a761932e721e1bc73394\": container with ID starting with 730bc018d3138d424b7ee1d4dffdf25b9a8c4c03fe21a761932e721e1bc73394 not found: ID does not exist" containerID="730bc018d3138d424b7ee1d4dffdf25b9a8c4c03fe21a761932e721e1bc73394" Oct 31 13:52:33.459333 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:33.459263 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730bc018d3138d424b7ee1d4dffdf25b9a8c4c03fe21a761932e721e1bc73394"} err="failed to get container status \"730bc018d3138d424b7ee1d4dffdf25b9a8c4c03fe21a761932e721e1bc73394\": rpc error: code = NotFound desc = could not find container \"730bc018d3138d424b7ee1d4dffdf25b9a8c4c03fe21a761932e721e1bc73394\": container with ID starting with 730bc018d3138d424b7ee1d4dffdf25b9a8c4c03fe21a761932e721e1bc73394 not found: ID does not exist" Oct 31 13:52:33.492542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:33.492513 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz"] Oct 31 13:52:33.498925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:33.498899 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-dlwlz"] Oct 31 13:52:33.719897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:33.719822 5532 patch_prober.go:29] interesting pod/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:52:33.719897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:33.719872 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:52:34.029666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:34.029142 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75407a66-4806-493d-a535-bfaaaddfbe16" path="/var/lib/kubelet/pods/75407a66-4806-493d-a535-bfaaaddfbe16/volumes" Oct 31 13:52:36.960619 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:52:36.960560 5532 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb572ef9e_7dc1_4886_bfa4_ec84b98dee31.slice/crio-9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:52:37.019258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.019229 5532 patch_prober.go:29] interesting pod/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body= Oct 31 13:52:37.019368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.019292 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" Oct 31 13:52:37.169433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.169378 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:52:37.215725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.215692 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istiod-ca-cert\") pod \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " Oct 31 13:52:37.215905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.215743 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-kube-api-access-gwgll\" (UniqueName: \"kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-kube-api-access-gwgll\") pod \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " Oct 31 13:52:37.215905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.215883 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-data\") pod \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " Oct 31 13:52:37.216049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.215957 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-socket\") pod \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " Oct 31 13:52:37.216049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.215988 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-credential-socket\") pod \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " Oct 31 13:52:37.216170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.216078 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istiod-ca-cert" pod "b572ef9e-7dc1-4886-bfa4-ec84b98dee31" (UID: "b572ef9e-7dc1-4886-bfa4-ec84b98dee31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:52:37.216230 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.216190 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-envoy\") pod \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " Oct 31 13:52:37.216971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.216306 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-certs\") pod \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " Oct 31 13:52:37.216971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.216367 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-token\" (UniqueName: \"kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-token\") pod \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " Oct 31 13:52:37.216971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.216307 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-data" pod "b572ef9e-7dc1-4886-bfa4-ec84b98dee31" (UID: "b572ef9e-7dc1-4886-bfa4-ec84b98dee31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:37.216971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.216294 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-socket" pod "b572ef9e-7dc1-4886-bfa4-ec84b98dee31" (UID: "b572ef9e-7dc1-4886-bfa4-ec84b98dee31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:37.216971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.216402 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-podinfo\") pod \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\" (UID: \"b572ef9e-7dc1-4886-bfa4-ec84b98dee31\") " Oct 31 13:52:37.216971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.216525 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-certs" pod "b572ef9e-7dc1-4886-bfa4-ec84b98dee31" (UID: "b572ef9e-7dc1-4886-bfa4-ec84b98dee31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:37.216971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.216952 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-credential-socket" pod "b572ef9e-7dc1-4886-bfa4-ec84b98dee31" (UID: "b572ef9e-7dc1-4886-bfa4-ec84b98dee31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:37.217440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.217257 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:37.217440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.217277 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:37.217440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.217292 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:37.217440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.217308 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:37.217440 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.217322 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:37.219114 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.219086 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-podinfo" pod "b572ef9e-7dc1-4886-bfa4-ec84b98dee31" (UID: "b572ef9e-7dc1-4886-bfa4-ec84b98dee31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:52:37.219921 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.219896 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-kube-api-access-gwgll" pod "b572ef9e-7dc1-4886-bfa4-ec84b98dee31" (UID: "b572ef9e-7dc1-4886-bfa4-ec84b98dee31"). InnerVolumeSpecName "kube-api-access-gwgll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:52:37.220136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.220117 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-envoy" pod "b572ef9e-7dc1-4886-bfa4-ec84b98dee31" (UID: "b572ef9e-7dc1-4886-bfa4-ec84b98dee31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:37.230671 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.230644 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-token" pod "b572ef9e-7dc1-4886-bfa4-ec84b98dee31" (UID: "b572ef9e-7dc1-4886-bfa4-ec84b98dee31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:52:37.319134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.319101 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:37.319134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.319131 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:37.319134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.319141 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwgll\" (UniqueName: \"kubernetes.io/projected/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-kube-api-access-gwgll\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:37.319134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.319151 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b572ef9e-7dc1-4886-bfa4-ec84b98dee31-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:37.482210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.482122 5532 generic.go:356] "Generic (PLEG): container finished" podID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerID="9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa" exitCode=0 Oct 31 13:52:37.482210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.482173 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" Oct 31 13:52:37.482442 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.482208 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" event={"ID":"b572ef9e-7dc1-4886-bfa4-ec84b98dee31","Type":"ContainerDied","Data":"9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa"} Oct 31 13:52:37.482442 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.482289 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85" event={"ID":"b572ef9e-7dc1-4886-bfa4-ec84b98dee31","Type":"ContainerDied","Data":"336448b0a4f630e02668ef7e1ca83ba4df808f541fd20edcd6d0133a0ab23533"} Oct 31 13:52:37.482442 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.482312 5532 scope.go:122] "RemoveContainer" containerID="9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa" Oct 31 13:52:37.493755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.493735 5532 scope.go:122] "RemoveContainer" containerID="9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa" Oct 31 13:52:37.494035 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:52:37.494019 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa\": container with ID starting with 9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa not found: ID does not exist" containerID="9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa" Oct 31 13:52:37.494083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.494043 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa"} err="failed to get container status \"9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa\": rpc error: code = NotFound desc = could not find container \"9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa\": container with ID starting with 9b8c959dcc06563cc7cd78c710304637c2a854c21349949a6cd4635bc2ee4baa not found: ID does not exist" Oct 31 13:52:37.510542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.510514 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85"] Oct 31 13:52:37.514107 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:37.514090 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-aabeef-openshift-default-6b8d7f885c-gqd85"] Oct 31 13:52:38.005467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.005432 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" path="/var/lib/kubelet/pods/b572ef9e-7dc1-4886-bfa4-ec84b98dee31/volumes" Oct 31 13:52:38.822383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.821967 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:52:38.844003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.843965 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-podinfo\") pod \"60eb6552-2f31-4bd4-bfd7-9286e7620151\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " Oct 31 13:52:38.844227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.844090 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60eb6552-2f31-4bd4-bfd7-9286e7620151-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60eb6552-2f31-4bd4-bfd7-9286e7620151-istiod-ca-cert\") pod \"60eb6552-2f31-4bd4-bfd7-9286e7620151\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " Oct 31 13:52:38.844227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.844144 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-data\") pod \"60eb6552-2f31-4bd4-bfd7-9286e7620151\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " Oct 31 13:52:38.844227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.844216 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-credential-socket\") pod \"60eb6552-2f31-4bd4-bfd7-9286e7620151\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " Oct 31 13:52:38.844460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.844308 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-kube-api-access-zskqs\" (UniqueName: \"kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-kube-api-access-zskqs\") pod \"60eb6552-2f31-4bd4-bfd7-9286e7620151\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " Oct 31 13:52:38.844460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.844344 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-certs\") pod \"60eb6552-2f31-4bd4-bfd7-9286e7620151\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " Oct 31 13:52:38.844563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.844497 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-socket\") pod \"60eb6552-2f31-4bd4-bfd7-9286e7620151\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " Oct 31 13:52:38.845260 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.844770 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-envoy\") pod \"60eb6552-2f31-4bd4-bfd7-9286e7620151\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " Oct 31 13:52:38.845260 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.844946 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-token\" (UniqueName: \"kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-token\") pod \"60eb6552-2f31-4bd4-bfd7-9286e7620151\" (UID: \"60eb6552-2f31-4bd4-bfd7-9286e7620151\") " Oct 31 13:52:38.845553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.845455 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-socket" pod "60eb6552-2f31-4bd4-bfd7-9286e7620151" (UID: "60eb6552-2f31-4bd4-bfd7-9286e7620151"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:38.849502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.845756 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:38.849502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.846009 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-certs" pod "60eb6552-2f31-4bd4-bfd7-9286e7620151" (UID: "60eb6552-2f31-4bd4-bfd7-9286e7620151"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:38.849502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.846332 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60eb6552-2f31-4bd4-bfd7-9286e7620151-istiod-ca-cert" pod "60eb6552-2f31-4bd4-bfd7-9286e7620151" (UID: "60eb6552-2f31-4bd4-bfd7-9286e7620151"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:52:38.849502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.847836 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-data" pod "60eb6552-2f31-4bd4-bfd7-9286e7620151" (UID: "60eb6552-2f31-4bd4-bfd7-9286e7620151"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:38.849502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.848714 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-kube-api-access-zskqs" pod "60eb6552-2f31-4bd4-bfd7-9286e7620151" (UID: "60eb6552-2f31-4bd4-bfd7-9286e7620151"). InnerVolumeSpecName "kube-api-access-zskqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:52:38.849502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.848973 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-credential-socket" pod "60eb6552-2f31-4bd4-bfd7-9286e7620151" (UID: "60eb6552-2f31-4bd4-bfd7-9286e7620151"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:38.850396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.850371 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-envoy" pod "60eb6552-2f31-4bd4-bfd7-9286e7620151" (UID: "60eb6552-2f31-4bd4-bfd7-9286e7620151"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:52:38.850513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.850418 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-token" pod "60eb6552-2f31-4bd4-bfd7-9286e7620151" (UID: "60eb6552-2f31-4bd4-bfd7-9286e7620151"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:52:38.850577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.850546 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-podinfo" pod "60eb6552-2f31-4bd4-bfd7-9286e7620151" (UID: "60eb6552-2f31-4bd4-bfd7-9286e7620151"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:52:38.946859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.946827 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:38.946859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.946859 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:38.947082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.946871 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60eb6552-2f31-4bd4-bfd7-9286e7620151-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:38.947082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.946883 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:38.947082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.946895 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:38.947082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.946907 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zskqs\" (UniqueName: \"kubernetes.io/projected/60eb6552-2f31-4bd4-bfd7-9286e7620151-kube-api-access-zskqs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:38.947082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.946921 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:38.947082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:38.946935 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60eb6552-2f31-4bd4-bfd7-9286e7620151-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:52:39.501131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.501088 5532 generic.go:356] "Generic (PLEG): container finished" podID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerID="aa7d56d5bddbe82f00fbb10b4cc4609b710e4d4835da81b8b6bb7877be858444" exitCode=0 Oct 31 13:52:39.501561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.501144 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" Oct 31 13:52:39.501561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.501182 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" event={"ID":"60eb6552-2f31-4bd4-bfd7-9286e7620151","Type":"ContainerDied","Data":"aa7d56d5bddbe82f00fbb10b4cc4609b710e4d4835da81b8b6bb7877be858444"} Oct 31 13:52:39.501561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.501240 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz" event={"ID":"60eb6552-2f31-4bd4-bfd7-9286e7620151","Type":"ContainerDied","Data":"b79935bc111ba4c6bb25524b1111a304a9f0b1de43d5df1139bbabd629562ebe"} Oct 31 13:52:39.501561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.501262 5532 scope.go:122] "RemoveContainer" containerID="aa7d56d5bddbe82f00fbb10b4cc4609b710e4d4835da81b8b6bb7877be858444" Oct 31 13:52:39.513378 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.513360 5532 scope.go:122] "RemoveContainer" containerID="c3432d0d9aa6508df3debcab6523d69add315abc4d78c18e7e715c63ebe92522" Oct 31 13:52:39.523678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.523659 5532 scope.go:122] "RemoveContainer" containerID="aa7d56d5bddbe82f00fbb10b4cc4609b710e4d4835da81b8b6bb7877be858444" Oct 31 13:52:39.523952 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:52:39.523933 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa7d56d5bddbe82f00fbb10b4cc4609b710e4d4835da81b8b6bb7877be858444\": container with ID starting with aa7d56d5bddbe82f00fbb10b4cc4609b710e4d4835da81b8b6bb7877be858444 not found: ID does not exist" containerID="aa7d56d5bddbe82f00fbb10b4cc4609b710e4d4835da81b8b6bb7877be858444" Oct 31 13:52:39.524010 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.523961 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa7d56d5bddbe82f00fbb10b4cc4609b710e4d4835da81b8b6bb7877be858444"} err="failed to get container status \"aa7d56d5bddbe82f00fbb10b4cc4609b710e4d4835da81b8b6bb7877be858444\": rpc error: code = NotFound desc = could not find container \"aa7d56d5bddbe82f00fbb10b4cc4609b710e4d4835da81b8b6bb7877be858444\": container with ID starting with aa7d56d5bddbe82f00fbb10b4cc4609b710e4d4835da81b8b6bb7877be858444 not found: ID does not exist" Oct 31 13:52:39.524010 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.523983 5532 scope.go:122] "RemoveContainer" containerID="c3432d0d9aa6508df3debcab6523d69add315abc4d78c18e7e715c63ebe92522" Oct 31 13:52:39.524244 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:52:39.524217 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3432d0d9aa6508df3debcab6523d69add315abc4d78c18e7e715c63ebe92522\": container with ID starting with c3432d0d9aa6508df3debcab6523d69add315abc4d78c18e7e715c63ebe92522 not found: ID does not exist" containerID="c3432d0d9aa6508df3debcab6523d69add315abc4d78c18e7e715c63ebe92522" Oct 31 13:52:39.524350 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.524248 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3432d0d9aa6508df3debcab6523d69add315abc4d78c18e7e715c63ebe92522"} err="failed to get container status \"c3432d0d9aa6508df3debcab6523d69add315abc4d78c18e7e715c63ebe92522\": rpc error: code = NotFound desc = could not find container \"c3432d0d9aa6508df3debcab6523d69add315abc4d78c18e7e715c63ebe92522\": container with ID starting with c3432d0d9aa6508df3debcab6523d69add315abc4d78c18e7e715c63ebe92522 not found: ID does not exist" Oct 31 13:52:39.531947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.531923 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz"] Oct 31 13:52:39.537533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.537512 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-72c210-openshift-default-5c9fc9cb8c-nf9hz"] Oct 31 13:52:39.923707 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.923675 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5"] Oct 31 13:52:39.924402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924388 5532 cpu_manager.go:414] "containerMap: removing container" podUID="75407a66-4806-493d-a535-bfaaaddfbe16" containerName="istio-proxy" Oct 31 13:52:39.924457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924403 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75407a66-4806-493d-a535-bfaaaddfbe16" containerName="istio-proxy" Oct 31 13:52:39.924457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924413 5532 cpu_manager.go:414] "containerMap: removing container" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" Oct 31 13:52:39.924457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924418 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" Oct 31 13:52:39.924457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924439 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f548adc7-0a93-47df-b5b6-d2084ddc3111" containerName="revoke-keys" Oct 31 13:52:39.924457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924447 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f548adc7-0a93-47df-b5b6-d2084ddc3111" containerName="revoke-keys" Oct 31 13:52:39.924457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924458 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerName="istio-proxy" Oct 31 13:52:39.924699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924463 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerName="istio-proxy" Oct 31 13:52:39.924699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924487 5532 cpu_manager.go:414] "containerMap: removing container" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" Oct 31 13:52:39.924699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924491 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" Oct 31 13:52:39.924699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924608 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f548adc7-0a93-47df-b5b6-d2084ddc3111" containerName="revoke-keys" Oct 31 13:52:39.924699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924618 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b572ef9e-7dc1-4886-bfa4-ec84b98dee31" containerName="istio-proxy" Oct 31 13:52:39.924699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924628 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" Oct 31 13:52:39.924699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.924638 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="75407a66-4806-493d-a535-bfaaaddfbe16" containerName="istio-proxy" Oct 31 13:52:39.929846 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.929826 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" Oct 31 13:52:39.938364 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.938061 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-20b8a4\"" Oct 31 13:52:39.941416 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.940874 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5"] Oct 31 13:52:39.964031 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.964002 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvxqx\" (UniqueName: \"kubernetes.io/projected/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-kube-api-access-hvxqx\") pod \"payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5\" (UID: \"dd0271dc-44bc-41fc-9e77-a882c4fddd2e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" Oct 31 13:52:39.964333 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.964306 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5\" (UID: \"dd0271dc-44bc-41fc-9e77-a882c4fddd2e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" Oct 31 13:52:39.974653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.974621 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2"] Oct 31 13:52:39.976068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.976046 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" containerName="istio-proxy" Oct 31 13:52:39.980726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.980707 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" Oct 31 13:52:39.984917 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.984891 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2"] Oct 31 13:52:39.991046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:39.991026 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-20b8a4-dockercfg-tf4th\"" Oct 31 13:52:40.022458 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.022420 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60eb6552-2f31-4bd4-bfd7-9286e7620151" path="/var/lib/kubelet/pods/60eb6552-2f31-4bd4-bfd7-9286e7620151/volumes" Oct 31 13:52:40.120622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.118961 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd8s4\" (UniqueName: \"kubernetes.io/projected/1e663abe-09aa-4060-b7ad-dc235442ad97-kube-api-access-wd8s4\") pod \"payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2\" (UID: \"1e663abe-09aa-4060-b7ad-dc235442ad97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" Oct 31 13:52:40.120622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.119041 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5\" (UID: \"dd0271dc-44bc-41fc-9e77-a882c4fddd2e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" Oct 31 13:52:40.120622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.119257 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvxqx\" (UniqueName: \"kubernetes.io/projected/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-kube-api-access-hvxqx\") pod \"payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5\" (UID: \"dd0271dc-44bc-41fc-9e77-a882c4fddd2e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" Oct 31 13:52:40.120622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.119306 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e663abe-09aa-4060-b7ad-dc235442ad97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2\" (UID: \"1e663abe-09aa-4060-b7ad-dc235442ad97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" Oct 31 13:52:40.122515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.121858 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5\" (UID: \"dd0271dc-44bc-41fc-9e77-a882c4fddd2e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" Oct 31 13:52:40.170044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.170010 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvxqx\" (UniqueName: \"kubernetes.io/projected/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-kube-api-access-hvxqx\") pod \"payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5\" (UID: \"dd0271dc-44bc-41fc-9e77-a882c4fddd2e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" Oct 31 13:52:40.221541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.221458 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8s4\" (UniqueName: \"kubernetes.io/projected/1e663abe-09aa-4060-b7ad-dc235442ad97-kube-api-access-wd8s4\") pod \"payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2\" (UID: \"1e663abe-09aa-4060-b7ad-dc235442ad97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" Oct 31 13:52:40.221924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.221903 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e663abe-09aa-4060-b7ad-dc235442ad97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2\" (UID: \"1e663abe-09aa-4060-b7ad-dc235442ad97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" Oct 31 13:52:40.223165 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.223139 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e663abe-09aa-4060-b7ad-dc235442ad97-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2\" (UID: \"1e663abe-09aa-4060-b7ad-dc235442ad97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" Oct 31 13:52:40.237049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.237024 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8s4\" (UniqueName: \"kubernetes.io/projected/1e663abe-09aa-4060-b7ad-dc235442ad97-kube-api-access-wd8s4\") pod \"payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2\" (UID: \"1e663abe-09aa-4060-b7ad-dc235442ad97\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" Oct 31 13:52:40.247289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.247266 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" Oct 31 13:52:40.325243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.319201 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" Oct 31 13:52:40.541615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.541564 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5"] Oct 31 13:52:40.543341 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:52:40.543312 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd0271dc_44bc_41fc_9e77_a882c4fddd2e.slice/crio-796bcb9b80ab0a1d328078e33af8afa4dd00182d11ef9aade195b804f5867304 WatchSource:0}: Error finding container 796bcb9b80ab0a1d328078e33af8afa4dd00182d11ef9aade195b804f5867304: Status 404 returned error can't find the container with id 796bcb9b80ab0a1d328078e33af8afa4dd00182d11ef9aade195b804f5867304 Oct 31 13:52:40.628289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:40.628262 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2"] Oct 31 13:52:41.521856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:41.521818 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" event={"ID":"1e663abe-09aa-4060-b7ad-dc235442ad97","Type":"ContainerStarted","Data":"9f58a5b7cf4af31d2a891e01d8db2037e972248e0a0edb6ff7f616a3fa380db2"} Oct 31 13:52:41.522072 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:41.521888 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" event={"ID":"1e663abe-09aa-4060-b7ad-dc235442ad97","Type":"ContainerStarted","Data":"d57d95b56afedf92d2be039a8c6573d9c46ddd3caf2fa96e8edb311e54b2dcbe"} Oct 31 13:52:41.522072 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:41.521921 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" Oct 31 13:52:41.523896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:41.523864 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" event={"ID":"dd0271dc-44bc-41fc-9e77-a882c4fddd2e","Type":"ContainerStarted","Data":"118ee0b5d079b2a2a0e4e5fde11c178bdc0d10746500ac042d9ce038fd4f9026"} Oct 31 13:52:41.524042 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:41.523919 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" event={"ID":"dd0271dc-44bc-41fc-9e77-a882c4fddd2e","Type":"ContainerStarted","Data":"796bcb9b80ab0a1d328078e33af8afa4dd00182d11ef9aade195b804f5867304"} Oct 31 13:52:41.524042 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:41.523946 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" Oct 31 13:52:41.558956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:41.558894 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" podStartSLOduration=2.558876198 podStartE2EDuration="2.558876198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:52:41.553026666 +0000 UTC m=+2929.698770495" watchObservedRunningTime="2026-10-31 13:52:41.558876198 +0000 UTC m=+2929.704620028" Oct 31 13:52:41.577262 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:41.577202 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" podStartSLOduration=2.577188044 podStartE2EDuration="2.577188044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:52:41.574145161 +0000 UTC m=+2929.719888990" watchObservedRunningTime="2026-10-31 13:52:41.577188044 +0000 UTC m=+2929.722931872" Oct 31 13:52:52.253636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:52.253586 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:52:52.536219 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:52.536136 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" Oct 31 13:52:52.536539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:52.536520 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" Oct 31 13:52:55.069104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:55.067500 5532 scope.go:122] "RemoveContainer" containerID="cdb8221749a137a15b3a7509a39ba266e6a43e841512c34a16ae8736afe31bd2" Oct 31 13:52:55.081240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:55.081215 5532 scope.go:122] "RemoveContainer" containerID="0dac29201a1cf7747039682b256b93147257f30f7f6c6f27e7e203fe5f0a564a" Oct 31 13:52:55.093349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:55.093327 5532 scope.go:122] "RemoveContainer" containerID="afe623623c3943d0770451baab3609b8a6026c2af012cfb2919f43b9e9f82247" Oct 31 13:52:55.103324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:55.103305 5532 scope.go:122] "RemoveContainer" containerID="8218824cc36e51b2dbf919fecff6a9903dd70aa2f944acb5fec2873bab9feee8" Oct 31 13:52:55.113156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:55.113136 5532 scope.go:122] "RemoveContainer" containerID="8ec376f9dd5a039996cafa559ee13d45705dbe32523aab05fdcf2dc5a1562157" Oct 31 13:52:57.203946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.203911 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4"] Oct 31 13:52:57.210388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.210364 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" Oct 31 13:52:57.215097 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.215069 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:52:57.216386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.216334 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:52:57.218464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.218440 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4"] Oct 31 13:52:57.331972 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.331930 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a27b68e-47sr4\" (UID: \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" Oct 31 13:52:57.332179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.331986 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a27b68e-47sr4\" (UID: \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" Oct 31 13:52:57.332275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.332253 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdhpv\" (UniqueName: \"kubernetes.io/projected/e831ca3a-c7e0-4677-91c3-d50022c2fe32-kube-api-access-sdhpv\") pod \"maas-api-revoke-sub-3a27b68e-47sr4\" (UID: \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" Oct 31 13:52:57.441476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.437005 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdhpv\" (UniqueName: \"kubernetes.io/projected/e831ca3a-c7e0-4677-91c3-d50022c2fe32-kube-api-access-sdhpv\") pod \"maas-api-revoke-sub-3a27b68e-47sr4\" (UID: \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" Oct 31 13:52:57.441476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.437231 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a27b68e-47sr4\" (UID: \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" Oct 31 13:52:57.441476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.437286 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a27b68e-47sr4\" (UID: \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" Oct 31 13:52:57.441476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.439749 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a27b68e-47sr4\" (UID: \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" Oct 31 13:52:57.471660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.470441 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a27b68e-47sr4\" (UID: \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" Oct 31 13:52:57.477618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.477110 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdhpv\" (UniqueName: \"kubernetes.io/projected/e831ca3a-c7e0-4677-91c3-d50022c2fe32-kube-api-access-sdhpv\") pod \"maas-api-revoke-sub-3a27b68e-47sr4\" (UID: \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" Oct 31 13:52:57.527068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.527034 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" Oct 31 13:52:57.819882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:57.819851 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4"] Oct 31 13:52:57.820416 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:52:57.820382 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode831ca3a_c7e0_4677_91c3_d50022c2fe32.slice/crio-361acda067995664a1222cd8a5ad17a99e46f1e76d3d97d04d19fa728a138997 WatchSource:0}: Error finding container 361acda067995664a1222cd8a5ad17a99e46f1e76d3d97d04d19fa728a138997: Status 404 returned error can't find the container with id 361acda067995664a1222cd8a5ad17a99e46f1e76d3d97d04d19fa728a138997 Oct 31 13:52:58.740056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:58.740011 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" event={"ID":"e831ca3a-c7e0-4677-91c3-d50022c2fe32","Type":"ContainerStarted","Data":"2c114233e421efb74d2da78296ae1eb5869c46ff7b9e08c6c71b483105d4ef46"} Oct 31 13:52:58.740465 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:58.740079 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" event={"ID":"e831ca3a-c7e0-4677-91c3-d50022c2fe32","Type":"ContainerStarted","Data":"361acda067995664a1222cd8a5ad17a99e46f1e76d3d97d04d19fa728a138997"} Oct 31 13:52:58.780948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:58.780877 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" podStartSLOduration=1.7808506830000002 podStartE2EDuration="1.780850683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:52:58.768577986 +0000 UTC m=+2946.914321814" watchObservedRunningTime="2026-10-31 13:52:58.780850683 +0000 UTC m=+2946.926594512" Oct 31 13:52:59.749417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:59.749382 5532 generic.go:356] "Generic (PLEG): container finished" podID="e831ca3a-c7e0-4677-91c3-d50022c2fe32" containerID="2c114233e421efb74d2da78296ae1eb5869c46ff7b9e08c6c71b483105d4ef46" exitCode=0 Oct 31 13:52:59.749878 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:52:59.749461 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" event={"ID":"e831ca3a-c7e0-4677-91c3-d50022c2fe32","Type":"ContainerDied","Data":"2c114233e421efb74d2da78296ae1eb5869c46ff7b9e08c6c71b483105d4ef46"} Oct 31 13:53:00.933782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:00.933756 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" Oct 31 13:53:01.001750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.001715 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-xks-ca\") pod \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\" (UID: \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\") " Oct 31 13:53:01.002000 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.001908 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e831ca3a-c7e0-4677-91c3-d50022c2fe32-kube-api-access-sdhpv\" (UniqueName: \"kubernetes.io/projected/e831ca3a-c7e0-4677-91c3-d50022c2fe32-kube-api-access-sdhpv\") pod \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\" (UID: \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\") " Oct 31 13:53:01.002000 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.001979 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-openshift-ca\") pod \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\" (UID: \"e831ca3a-c7e0-4677-91c3-d50022c2fe32\") " Oct 31 13:53:01.002370 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.002337 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-openshift-ca" pod "e831ca3a-c7e0-4677-91c3-d50022c2fe32" (UID: "e831ca3a-c7e0-4677-91c3-d50022c2fe32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:01.002772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.002744 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:01.003967 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.003943 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-xks-ca" pod "e831ca3a-c7e0-4677-91c3-d50022c2fe32" (UID: "e831ca3a-c7e0-4677-91c3-d50022c2fe32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:53:01.004221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.004204 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e831ca3a-c7e0-4677-91c3-d50022c2fe32-kube-api-access-sdhpv" pod "e831ca3a-c7e0-4677-91c3-d50022c2fe32" (UID: "e831ca3a-c7e0-4677-91c3-d50022c2fe32"). InnerVolumeSpecName "kube-api-access-sdhpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:01.104388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.104188 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdhpv\" (UniqueName: \"kubernetes.io/projected/e831ca3a-c7e0-4677-91c3-d50022c2fe32-kube-api-access-sdhpv\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:01.104388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.104229 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e831ca3a-c7e0-4677-91c3-d50022c2fe32-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:01.351934 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.351851 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg"] Oct 31 13:53:01.352620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.352587 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e831ca3a-c7e0-4677-91c3-d50022c2fe32" containerName="revoke-keys" Oct 31 13:53:01.352674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.352623 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e831ca3a-c7e0-4677-91c3-d50022c2fe32" containerName="revoke-keys" Oct 31 13:53:01.352854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.352843 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e831ca3a-c7e0-4677-91c3-d50022c2fe32" containerName="revoke-keys" Oct 31 13:53:01.356878 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.356855 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" Oct 31 13:53:01.366708 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.366605 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg"] Oct 31 13:53:01.410703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.410663 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84fa10ce-vtpsg\" (UID: \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" Oct 31 13:53:01.410995 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.410975 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4rgt\" (UniqueName: \"kubernetes.io/projected/66852bb5-e526-4dac-a82a-a986ad2cbd5a-kube-api-access-m4rgt\") pod \"maas-api-revoke-sub-84fa10ce-vtpsg\" (UID: \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" Oct 31 13:53:01.411097 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.411048 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84fa10ce-vtpsg\" (UID: \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" Oct 31 13:53:01.512744 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.512709 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84fa10ce-vtpsg\" (UID: \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" Oct 31 13:53:01.513000 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.512980 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4rgt\" (UniqueName: \"kubernetes.io/projected/66852bb5-e526-4dac-a82a-a986ad2cbd5a-kube-api-access-m4rgt\") pod \"maas-api-revoke-sub-84fa10ce-vtpsg\" (UID: \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" Oct 31 13:53:01.513105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.513061 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84fa10ce-vtpsg\" (UID: \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" Oct 31 13:53:01.513749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.513727 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84fa10ce-vtpsg\" (UID: \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" Oct 31 13:53:01.515083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.515060 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84fa10ce-vtpsg\" (UID: \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" Oct 31 13:53:01.527027 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.527004 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4rgt\" (UniqueName: \"kubernetes.io/projected/66852bb5-e526-4dac-a82a-a986ad2cbd5a-kube-api-access-m4rgt\") pod \"maas-api-revoke-sub-84fa10ce-vtpsg\" (UID: \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" Oct 31 13:53:01.674366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.674336 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" Oct 31 13:53:01.777912 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:53:01.777199 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:53:01.779301 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.779274 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" event={"ID":"e831ca3a-c7e0-4677-91c3-d50022c2fe32","Type":"ContainerDied","Data":"361acda067995664a1222cd8a5ad17a99e46f1e76d3d97d04d19fa728a138997"} Oct 31 13:53:01.779424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.779335 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="361acda067995664a1222cd8a5ad17a99e46f1e76d3d97d04d19fa728a138997" Oct 31 13:53:01.779424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.779405 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4" Oct 31 13:53:01.964757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:01.964718 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg"] Oct 31 13:53:01.965505 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:53:01.965481 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66852bb5_e526_4dac_a82a_a986ad2cbd5a.slice/crio-3225f3b805daeb71d556d0fe31d1ef1904a68c2fbe225c80c9eda43f7e32812b WatchSource:0}: Error finding container 3225f3b805daeb71d556d0fe31d1ef1904a68c2fbe225c80c9eda43f7e32812b: Status 404 returned error can't find the container with id 3225f3b805daeb71d556d0fe31d1ef1904a68c2fbe225c80c9eda43f7e32812b Oct 31 13:53:02.789303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:02.789260 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" event={"ID":"66852bb5-e526-4dac-a82a-a986ad2cbd5a","Type":"ContainerStarted","Data":"aafd281455ecedc32bda4bb05721832e347337c81d403c124cb5461892d26de6"} Oct 31 13:53:02.789512 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:02.789326 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" event={"ID":"66852bb5-e526-4dac-a82a-a986ad2cbd5a","Type":"ContainerStarted","Data":"3225f3b805daeb71d556d0fe31d1ef1904a68c2fbe225c80c9eda43f7e32812b"} Oct 31 13:53:02.812583 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:02.812532 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" podStartSLOduration=1.812517825 podStartE2EDuration="1.812517825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:02.80956876 +0000 UTC m=+2950.955312589" watchObservedRunningTime="2026-10-31 13:53:02.812517825 +0000 UTC m=+2950.958261652" Oct 31 13:53:03.798365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:03.798330 5532 generic.go:356] "Generic (PLEG): container finished" podID="66852bb5-e526-4dac-a82a-a986ad2cbd5a" containerID="aafd281455ecedc32bda4bb05721832e347337c81d403c124cb5461892d26de6" exitCode=0 Oct 31 13:53:03.798777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:03.798406 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" event={"ID":"66852bb5-e526-4dac-a82a-a986ad2cbd5a","Type":"ContainerDied","Data":"aafd281455ecedc32bda4bb05721832e347337c81d403c124cb5461892d26de6"} Oct 31 13:53:04.970226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:04.970201 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" Oct 31 13:53:05.069847 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:05.069765 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-openshift-ca\") pod \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\" (UID: \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\") " Oct 31 13:53:05.069847 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:05.069814 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66852bb5-e526-4dac-a82a-a986ad2cbd5a-kube-api-access-m4rgt\" (UniqueName: \"kubernetes.io/projected/66852bb5-e526-4dac-a82a-a986ad2cbd5a-kube-api-access-m4rgt\") pod \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\" (UID: \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\") " Oct 31 13:53:05.070086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:05.069928 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-xks-ca\") pod \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\" (UID: \"66852bb5-e526-4dac-a82a-a986ad2cbd5a\") " Oct 31 13:53:05.070219 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:05.070193 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-openshift-ca" pod "66852bb5-e526-4dac-a82a-a986ad2cbd5a" (UID: "66852bb5-e526-4dac-a82a-a986ad2cbd5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:05.070457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:05.070438 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:05.071985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:05.071963 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-xks-ca" pod "66852bb5-e526-4dac-a82a-a986ad2cbd5a" (UID: "66852bb5-e526-4dac-a82a-a986ad2cbd5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:53:05.072075 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:05.072055 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66852bb5-e526-4dac-a82a-a986ad2cbd5a-kube-api-access-m4rgt" pod "66852bb5-e526-4dac-a82a-a986ad2cbd5a" (UID: "66852bb5-e526-4dac-a82a-a986ad2cbd5a"). InnerVolumeSpecName "kube-api-access-m4rgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:05.172334 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:05.172300 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66852bb5-e526-4dac-a82a-a986ad2cbd5a-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:05.172334 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:05.172332 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4rgt\" (UniqueName: \"kubernetes.io/projected/66852bb5-e526-4dac-a82a-a986ad2cbd5a-kube-api-access-m4rgt\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:05.815163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:05.815136 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" Oct 31 13:53:05.815163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:05.815146 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg" event={"ID":"66852bb5-e526-4dac-a82a-a986ad2cbd5a","Type":"ContainerDied","Data":"3225f3b805daeb71d556d0fe31d1ef1904a68c2fbe225c80c9eda43f7e32812b"} Oct 31 13:53:05.815403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:05.815196 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3225f3b805daeb71d556d0fe31d1ef1904a68c2fbe225c80c9eda43f7e32812b" Oct 31 13:53:07.429485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.429452 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9"] Oct 31 13:53:07.430206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.430189 5532 cpu_manager.go:414] "containerMap: removing container" podUID="66852bb5-e526-4dac-a82a-a986ad2cbd5a" containerName="revoke-keys" Oct 31 13:53:07.430267 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.430207 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66852bb5-e526-4dac-a82a-a986ad2cbd5a" containerName="revoke-keys" Oct 31 13:53:07.430375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.430365 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="66852bb5-e526-4dac-a82a-a986ad2cbd5a" containerName="revoke-keys" Oct 31 13:53:07.434388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.434373 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" Oct 31 13:53:07.440165 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.440145 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:53:07.441355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.441331 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:53:07.447092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.447069 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9"] Oct 31 13:53:07.510110 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.510076 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5lc5\" (UniqueName: \"kubernetes.io/projected/af844940-1e57-4c69-8c42-9e2dc349ae54-kube-api-access-w5lc5\") pod \"maas-api-revoke-sub-296ed273-t75z9\" (UID: \"af844940-1e57-4c69-8c42-9e2dc349ae54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" Oct 31 13:53:07.510319 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.510227 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-296ed273-t75z9\" (UID: \"af844940-1e57-4c69-8c42-9e2dc349ae54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" Oct 31 13:53:07.510402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.510384 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-296ed273-t75z9\" (UID: \"af844940-1e57-4c69-8c42-9e2dc349ae54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" Oct 31 13:53:07.611782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.611744 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-296ed273-t75z9\" (UID: \"af844940-1e57-4c69-8c42-9e2dc349ae54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" Oct 31 13:53:07.611960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.611915 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-296ed273-t75z9\" (UID: \"af844940-1e57-4c69-8c42-9e2dc349ae54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" Oct 31 13:53:07.612118 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.612100 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5lc5\" (UniqueName: \"kubernetes.io/projected/af844940-1e57-4c69-8c42-9e2dc349ae54-kube-api-access-w5lc5\") pod \"maas-api-revoke-sub-296ed273-t75z9\" (UID: \"af844940-1e57-4c69-8c42-9e2dc349ae54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" Oct 31 13:53:07.612544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.612523 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-296ed273-t75z9\" (UID: \"af844940-1e57-4c69-8c42-9e2dc349ae54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" Oct 31 13:53:07.614204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.614180 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-xks-ca\") pod \"maas-api-revoke-sub-296ed273-t75z9\" (UID: \"af844940-1e57-4c69-8c42-9e2dc349ae54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" Oct 31 13:53:07.626423 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.626395 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5lc5\" (UniqueName: \"kubernetes.io/projected/af844940-1e57-4c69-8c42-9e2dc349ae54-kube-api-access-w5lc5\") pod \"maas-api-revoke-sub-296ed273-t75z9\" (UID: \"af844940-1e57-4c69-8c42-9e2dc349ae54\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" Oct 31 13:53:07.752216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:07.752139 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" Oct 31 13:53:08.046701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:08.045605 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9"] Oct 31 13:53:08.055680 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:53:08.055649 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf844940_1e57_4c69_8c42_9e2dc349ae54.slice/crio-d7e8e137f9ab8b6d744649bb69836039814f94a6c1e6af468435ebe2b4bf5e08 WatchSource:0}: Error finding container d7e8e137f9ab8b6d744649bb69836039814f94a6c1e6af468435ebe2b4bf5e08: Status 404 returned error can't find the container with id d7e8e137f9ab8b6d744649bb69836039814f94a6c1e6af468435ebe2b4bf5e08 Oct 31 13:53:08.850529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:08.850483 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" event={"ID":"af844940-1e57-4c69-8c42-9e2dc349ae54","Type":"ContainerStarted","Data":"f6317c268489b26a5909ddc687d4f19ed37b79edee559072899bacbed18ad2e1"} Oct 31 13:53:08.851028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:08.850588 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" event={"ID":"af844940-1e57-4c69-8c42-9e2dc349ae54","Type":"ContainerStarted","Data":"d7e8e137f9ab8b6d744649bb69836039814f94a6c1e6af468435ebe2b4bf5e08"} Oct 31 13:53:08.873474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:08.873426 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" podStartSLOduration=1.873412063 podStartE2EDuration="1.873412063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:08.870769408 +0000 UTC m=+2957.016513239" watchObservedRunningTime="2026-10-31 13:53:08.873412063 +0000 UTC m=+2957.019155891" Oct 31 13:53:09.861509 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:09.861471 5532 generic.go:356] "Generic (PLEG): container finished" podID="af844940-1e57-4c69-8c42-9e2dc349ae54" containerID="f6317c268489b26a5909ddc687d4f19ed37b79edee559072899bacbed18ad2e1" exitCode=0 Oct 31 13:53:09.861927 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:09.861552 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" event={"ID":"af844940-1e57-4c69-8c42-9e2dc349ae54","Type":"ContainerDied","Data":"f6317c268489b26a5909ddc687d4f19ed37b79edee559072899bacbed18ad2e1"} Oct 31 13:53:11.052801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.052769 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" Oct 31 13:53:11.176922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.176891 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-xks-ca\") pod \"af844940-1e57-4c69-8c42-9e2dc349ae54\" (UID: \"af844940-1e57-4c69-8c42-9e2dc349ae54\") " Oct 31 13:53:11.177108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.177020 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-openshift-ca\") pod \"af844940-1e57-4c69-8c42-9e2dc349ae54\" (UID: \"af844940-1e57-4c69-8c42-9e2dc349ae54\") " Oct 31 13:53:11.177178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.177166 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af844940-1e57-4c69-8c42-9e2dc349ae54-kube-api-access-w5lc5\" (UniqueName: \"kubernetes.io/projected/af844940-1e57-4c69-8c42-9e2dc349ae54-kube-api-access-w5lc5\") pod \"af844940-1e57-4c69-8c42-9e2dc349ae54\" (UID: \"af844940-1e57-4c69-8c42-9e2dc349ae54\") " Oct 31 13:53:11.177391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.177362 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-openshift-ca" pod "af844940-1e57-4c69-8c42-9e2dc349ae54" (UID: "af844940-1e57-4c69-8c42-9e2dc349ae54"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:11.177794 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.177771 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:11.179019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.178993 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-xks-ca" pod "af844940-1e57-4c69-8c42-9e2dc349ae54" (UID: "af844940-1e57-4c69-8c42-9e2dc349ae54"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:53:11.179276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.179256 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af844940-1e57-4c69-8c42-9e2dc349ae54-kube-api-access-w5lc5" pod "af844940-1e57-4c69-8c42-9e2dc349ae54" (UID: "af844940-1e57-4c69-8c42-9e2dc349ae54"). InnerVolumeSpecName "kube-api-access-w5lc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:11.279361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.279329 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5lc5\" (UniqueName: \"kubernetes.io/projected/af844940-1e57-4c69-8c42-9e2dc349ae54-kube-api-access-w5lc5\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:11.279361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.279358 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af844940-1e57-4c69-8c42-9e2dc349ae54-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:11.596696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.596618 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r"] Oct 31 13:53:11.597389 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.597374 5532 cpu_manager.go:414] "containerMap: removing container" podUID="af844940-1e57-4c69-8c42-9e2dc349ae54" containerName="revoke-keys" Oct 31 13:53:11.597453 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.597390 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af844940-1e57-4c69-8c42-9e2dc349ae54" containerName="revoke-keys" Oct 31 13:53:11.597554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.597545 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="af844940-1e57-4c69-8c42-9e2dc349ae54" containerName="revoke-keys" Oct 31 13:53:11.601456 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.601441 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" Oct 31 13:53:11.612465 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.612438 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r"] Oct 31 13:53:11.800191 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.800156 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vct2\" (UniqueName: \"kubernetes.io/projected/5414af14-988d-4f31-9e65-479f5a6d07d7-kube-api-access-4vct2\") pod \"maas-api-revoke-sub-565b4540-9667r\" (UID: \"5414af14-988d-4f31-9e65-479f5a6d07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" Oct 31 13:53:11.800392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.800213 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-565b4540-9667r\" (UID: \"5414af14-988d-4f31-9e65-479f5a6d07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" Oct 31 13:53:11.800392 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.800244 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-565b4540-9667r\" (UID: \"5414af14-988d-4f31-9e65-479f5a6d07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" Oct 31 13:53:11.880420 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.880341 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" Oct 31 13:53:11.880420 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.880358 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9" event={"ID":"af844940-1e57-4c69-8c42-9e2dc349ae54","Type":"ContainerDied","Data":"d7e8e137f9ab8b6d744649bb69836039814f94a6c1e6af468435ebe2b4bf5e08"} Oct 31 13:53:11.880420 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.880411 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7e8e137f9ab8b6d744649bb69836039814f94a6c1e6af468435ebe2b4bf5e08" Oct 31 13:53:11.901883 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.901853 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vct2\" (UniqueName: \"kubernetes.io/projected/5414af14-988d-4f31-9e65-479f5a6d07d7-kube-api-access-4vct2\") pod \"maas-api-revoke-sub-565b4540-9667r\" (UID: \"5414af14-988d-4f31-9e65-479f5a6d07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" Oct 31 13:53:11.902034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.901902 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-565b4540-9667r\" (UID: \"5414af14-988d-4f31-9e65-479f5a6d07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" Oct 31 13:53:11.902034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.901930 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-565b4540-9667r\" (UID: \"5414af14-988d-4f31-9e65-479f5a6d07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" Oct 31 13:53:11.902552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.902522 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-565b4540-9667r\" (UID: \"5414af14-988d-4f31-9e65-479f5a6d07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" Oct 31 13:53:11.904343 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.904323 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-565b4540-9667r\" (UID: \"5414af14-988d-4f31-9e65-479f5a6d07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" Oct 31 13:53:11.914903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.914881 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vct2\" (UniqueName: \"kubernetes.io/projected/5414af14-988d-4f31-9e65-479f5a6d07d7-kube-api-access-4vct2\") pod \"maas-api-revoke-sub-565b4540-9667r\" (UID: \"5414af14-988d-4f31-9e65-479f5a6d07d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" Oct 31 13:53:11.918793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:11.918771 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" Oct 31 13:53:12.203657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:12.203626 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r"] Oct 31 13:53:12.204320 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:53:12.204295 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5414af14_988d_4f31_9e65_479f5a6d07d7.slice/crio-7aad02b1f34f02ad1773167a56d1c43efcfbd8c1b4211fc4f49a0f242640606c WatchSource:0}: Error finding container 7aad02b1f34f02ad1773167a56d1c43efcfbd8c1b4211fc4f49a0f242640606c: Status 404 returned error can't find the container with id 7aad02b1f34f02ad1773167a56d1c43efcfbd8c1b4211fc4f49a0f242640606c Oct 31 13:53:12.890024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:12.889925 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" event={"ID":"5414af14-988d-4f31-9e65-479f5a6d07d7","Type":"ContainerStarted","Data":"0ba6b770ae96e4759aa5cceaafae478de7e34f7bf0938463e4be39e8b8bcc573"} Oct 31 13:53:12.890024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:12.889994 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" event={"ID":"5414af14-988d-4f31-9e65-479f5a6d07d7","Type":"ContainerStarted","Data":"7aad02b1f34f02ad1773167a56d1c43efcfbd8c1b4211fc4f49a0f242640606c"} Oct 31 13:53:12.916685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:12.916620 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" podStartSLOduration=1.916603692 podStartE2EDuration="1.916603692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:12.911568607 +0000 UTC m=+2961.057312435" watchObservedRunningTime="2026-10-31 13:53:12.916603692 +0000 UTC m=+2961.062347514" Oct 31 13:53:13.899714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:13.899672 5532 generic.go:356] "Generic (PLEG): container finished" podID="5414af14-988d-4f31-9e65-479f5a6d07d7" containerID="0ba6b770ae96e4759aa5cceaafae478de7e34f7bf0938463e4be39e8b8bcc573" exitCode=0 Oct 31 13:53:13.900116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:13.899750 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" event={"ID":"5414af14-988d-4f31-9e65-479f5a6d07d7","Type":"ContainerDied","Data":"0ba6b770ae96e4759aa5cceaafae478de7e34f7bf0938463e4be39e8b8bcc573"} Oct 31 13:53:15.121528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:15.121502 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" Oct 31 13:53:15.170338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:15.170255 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5414af14-988d-4f31-9e65-479f5a6d07d7-kube-api-access-4vct2\" (UniqueName: \"kubernetes.io/projected/5414af14-988d-4f31-9e65-479f5a6d07d7-kube-api-access-4vct2\") pod \"5414af14-988d-4f31-9e65-479f5a6d07d7\" (UID: \"5414af14-988d-4f31-9e65-479f5a6d07d7\") " Oct 31 13:53:15.170338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:15.170303 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-xks-ca\") pod \"5414af14-988d-4f31-9e65-479f5a6d07d7\" (UID: \"5414af14-988d-4f31-9e65-479f5a6d07d7\") " Oct 31 13:53:15.170569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:15.170361 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-openshift-ca\") pod \"5414af14-988d-4f31-9e65-479f5a6d07d7\" (UID: \"5414af14-988d-4f31-9e65-479f5a6d07d7\") " Oct 31 13:53:15.170892 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:15.170857 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-openshift-ca" pod "5414af14-988d-4f31-9e65-479f5a6d07d7" (UID: "5414af14-988d-4f31-9e65-479f5a6d07d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:15.171133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:15.171109 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:15.172669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:15.172633 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5414af14-988d-4f31-9e65-479f5a6d07d7-kube-api-access-4vct2" pod "5414af14-988d-4f31-9e65-479f5a6d07d7" (UID: "5414af14-988d-4f31-9e65-479f5a6d07d7"). InnerVolumeSpecName "kube-api-access-4vct2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:15.172669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:15.172647 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-xks-ca" pod "5414af14-988d-4f31-9e65-479f5a6d07d7" (UID: "5414af14-988d-4f31-9e65-479f5a6d07d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:53:15.272860 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:15.272822 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vct2\" (UniqueName: \"kubernetes.io/projected/5414af14-988d-4f31-9e65-479f5a6d07d7-kube-api-access-4vct2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:15.272860 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:15.272856 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5414af14-988d-4f31-9e65-479f5a6d07d7-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:15.930637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:15.930554 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" event={"ID":"5414af14-988d-4f31-9e65-479f5a6d07d7","Type":"ContainerDied","Data":"7aad02b1f34f02ad1773167a56d1c43efcfbd8c1b4211fc4f49a0f242640606c"} Oct 31 13:53:15.930835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:15.930670 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aad02b1f34f02ad1773167a56d1c43efcfbd8c1b4211fc4f49a0f242640606c" Oct 31 13:53:15.930835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:15.930753 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r" Oct 31 13:53:16.971568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:16.971532 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476"] Oct 31 13:53:16.972037 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:16.971849 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerName="istio-proxy" containerID="cri-o://a2b7912fbf65c0403d50a81b01fe037857d9412d10d8b681789cb5788a102c52" gracePeriod=30 Oct 31 13:53:17.333983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:17.333872 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq"] Oct 31 13:53:17.334209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:17.334112 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerName="istio-proxy" containerID="cri-o://bb552b3a689ad9b0102b281738bbfb59b6a006990466e803857fe92c6d429510" gracePeriod=30 Oct 31 13:53:18.842657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:18.842590 5532 patch_prober.go:29] interesting pod/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:53:18.843052 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:18.842673 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:53:18.905421 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:18.905387 5532 patch_prober.go:29] interesting pod/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:53:18.905563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:18.905441 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:53:19.091251 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:19.091208 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5"] Oct 31 13:53:19.107013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:19.106941 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2"] Oct 31 13:53:22.282630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.282586 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:53:22.365301 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.365265 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-certs\") pod \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " Oct 31 13:53:22.365438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.365316 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-podinfo\") pod \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " Oct 31 13:53:22.365438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.365359 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-kube-api-access-wkhnw\" (UniqueName: \"kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-kube-api-access-wkhnw\") pod \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " Oct 31 13:53:22.365674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.365470 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-credential-socket\") pod \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " Oct 31 13:53:22.365674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.365517 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-socket\") pod \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " Oct 31 13:53:22.365674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.365639 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-token\" (UniqueName: \"kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-token\") pod \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " Oct 31 13:53:22.365856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.365665 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-certs" pod "8a7ef42d-5b18-414a-a0ff-492cc511c0db" (UID: "8a7ef42d-5b18-414a-a0ff-492cc511c0db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:22.365856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.365691 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-envoy\") pod \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " Oct 31 13:53:22.365856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.365753 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istiod-ca-cert\") pod \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " Oct 31 13:53:22.366020 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.365936 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-data\") pod \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\" (UID: \"8a7ef42d-5b18-414a-a0ff-492cc511c0db\") " Oct 31 13:53:22.366395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.366116 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-credential-socket" pod "8a7ef42d-5b18-414a-a0ff-492cc511c0db" (UID: "8a7ef42d-5b18-414a-a0ff-492cc511c0db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:22.366471 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.366449 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istiod-ca-cert" pod "8a7ef42d-5b18-414a-a0ff-492cc511c0db" (UID: "8a7ef42d-5b18-414a-a0ff-492cc511c0db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:22.366751 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.366727 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-data" pod "8a7ef42d-5b18-414a-a0ff-492cc511c0db" (UID: "8a7ef42d-5b18-414a-a0ff-492cc511c0db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:22.367054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.367030 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.367133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.367061 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.367133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.367076 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.367133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.367090 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.367317 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.367282 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-socket" pod "8a7ef42d-5b18-414a-a0ff-492cc511c0db" (UID: "8a7ef42d-5b18-414a-a0ff-492cc511c0db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:22.367946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.367919 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-podinfo" pod "8a7ef42d-5b18-414a-a0ff-492cc511c0db" (UID: "8a7ef42d-5b18-414a-a0ff-492cc511c0db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:53:22.368142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.368119 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-kube-api-access-wkhnw" pod "8a7ef42d-5b18-414a-a0ff-492cc511c0db" (UID: "8a7ef42d-5b18-414a-a0ff-492cc511c0db"). InnerVolumeSpecName "kube-api-access-wkhnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:22.368409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.368387 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-token" pod "8a7ef42d-5b18-414a-a0ff-492cc511c0db" (UID: "8a7ef42d-5b18-414a-a0ff-492cc511c0db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:22.368991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.368971 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-envoy" pod "8a7ef42d-5b18-414a-a0ff-492cc511c0db" (UID: "8a7ef42d-5b18-414a-a0ff-492cc511c0db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:22.468726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.468691 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkhnw\" (UniqueName: \"kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-kube-api-access-wkhnw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.468726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.468722 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.468726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.468732 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.469019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.468743 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.469019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.468752 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a7ef42d-5b18-414a-a0ff-492cc511c0db-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.637844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.637811 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:53:22.672085 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.672056 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-podinfo\") pod \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " Oct 31 13:53:22.672241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.672112 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-envoy\") pod \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " Oct 31 13:53:22.672241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.672203 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-data\") pod \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " Oct 31 13:53:22.672321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.672274 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-credential-socket\") pod \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " Oct 31 13:53:22.672375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.672340 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-socket\") pod \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " Oct 31 13:53:22.672491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.672469 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-kube-api-access-9bdbv\" (UniqueName: \"kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-kube-api-access-9bdbv\") pod \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " Oct 31 13:53:22.672577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.672522 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-token\" (UniqueName: \"kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-token\") pod \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " Oct 31 13:53:22.672651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.672638 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istiod-ca-cert\") pod \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " Oct 31 13:53:22.672754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.672734 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-certs\") pod \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\" (UID: \"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4\") " Oct 31 13:53:22.672883 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.672779 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-data" pod "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" (UID: "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:22.672883 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.672848 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-credential-socket" pod "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" (UID: "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:22.673157 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.673135 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istiod-ca-cert" pod "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" (UID: "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:22.673314 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.673264 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-socket" pod "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" (UID: "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:22.673389 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.673342 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-certs" pod "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" (UID: "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:22.673656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.673634 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.673656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.673655 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.673817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.673666 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.673817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.673675 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.673817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.673684 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.674674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.674635 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-envoy" pod "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" (UID: "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:22.674840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.674687 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-podinfo" pod "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" (UID: "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:53:22.675203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.675182 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-kube-api-access-9bdbv" pod "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" (UID: "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4"). InnerVolumeSpecName "kube-api-access-9bdbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:22.675794 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.675777 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-token" pod "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" (UID: "0ff1a789-0fbf-43ee-91dc-41eb7636c4e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:22.777514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.777418 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bdbv\" (UniqueName: \"kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-kube-api-access-9bdbv\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.777514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.777459 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.777514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.777475 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.777514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.777488 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:22.990960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.990927 5532 generic.go:356] "Generic (PLEG): container finished" podID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerID="a2b7912fbf65c0403d50a81b01fe037857d9412d10d8b681789cb5788a102c52" exitCode=0 Oct 31 13:53:22.991149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.990979 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" Oct 31 13:53:22.991149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.991017 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" event={"ID":"8a7ef42d-5b18-414a-a0ff-492cc511c0db","Type":"ContainerDied","Data":"a2b7912fbf65c0403d50a81b01fe037857d9412d10d8b681789cb5788a102c52"} Oct 31 13:53:22.991149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.991076 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476" event={"ID":"8a7ef42d-5b18-414a-a0ff-492cc511c0db","Type":"ContainerDied","Data":"d7417a7a68a3097f9eebb0b408a9b84dd216ce90895cccf413b434e8230f9865"} Oct 31 13:53:22.991149 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.991103 5532 scope.go:122] "RemoveContainer" containerID="a2b7912fbf65c0403d50a81b01fe037857d9412d10d8b681789cb5788a102c52" Oct 31 13:53:22.993279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.993252 5532 generic.go:356] "Generic (PLEG): container finished" podID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerID="bb552b3a689ad9b0102b281738bbfb59b6a006990466e803857fe92c6d429510" exitCode=0 Oct 31 13:53:22.993399 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.993312 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" Oct 31 13:53:22.993472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.993400 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" event={"ID":"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4","Type":"ContainerDied","Data":"bb552b3a689ad9b0102b281738bbfb59b6a006990466e803857fe92c6d429510"} Oct 31 13:53:22.993472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:22.993439 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq" event={"ID":"0ff1a789-0fbf-43ee-91dc-41eb7636c4e4","Type":"ContainerDied","Data":"cdd304301ae41d1c1444cc2dc871d8a0856ded81b217fc338aa16f7539afb767"} Oct 31 13:53:23.017329 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.017309 5532 scope.go:122] "RemoveContainer" containerID="dd02214c26380e2ef79dacdbf9badce4d57811fff29ca0ce19a075348c5146b4" Oct 31 13:53:23.027701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.027660 5532 scope.go:122] "RemoveContainer" containerID="a2b7912fbf65c0403d50a81b01fe037857d9412d10d8b681789cb5788a102c52" Oct 31 13:53:23.027952 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:53:23.027933 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2b7912fbf65c0403d50a81b01fe037857d9412d10d8b681789cb5788a102c52\": container with ID starting with a2b7912fbf65c0403d50a81b01fe037857d9412d10d8b681789cb5788a102c52 not found: ID does not exist" containerID="a2b7912fbf65c0403d50a81b01fe037857d9412d10d8b681789cb5788a102c52" Oct 31 13:53:23.028009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.027960 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2b7912fbf65c0403d50a81b01fe037857d9412d10d8b681789cb5788a102c52"} err="failed to get container status \"a2b7912fbf65c0403d50a81b01fe037857d9412d10d8b681789cb5788a102c52\": rpc error: code = NotFound desc = could not find container \"a2b7912fbf65c0403d50a81b01fe037857d9412d10d8b681789cb5788a102c52\": container with ID starting with a2b7912fbf65c0403d50a81b01fe037857d9412d10d8b681789cb5788a102c52 not found: ID does not exist" Oct 31 13:53:23.028009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.027981 5532 scope.go:122] "RemoveContainer" containerID="dd02214c26380e2ef79dacdbf9badce4d57811fff29ca0ce19a075348c5146b4" Oct 31 13:53:23.028220 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:53:23.028203 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd02214c26380e2ef79dacdbf9badce4d57811fff29ca0ce19a075348c5146b4\": container with ID starting with dd02214c26380e2ef79dacdbf9badce4d57811fff29ca0ce19a075348c5146b4 not found: ID does not exist" containerID="dd02214c26380e2ef79dacdbf9badce4d57811fff29ca0ce19a075348c5146b4" Oct 31 13:53:23.028264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.028227 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd02214c26380e2ef79dacdbf9badce4d57811fff29ca0ce19a075348c5146b4"} err="failed to get container status \"dd02214c26380e2ef79dacdbf9badce4d57811fff29ca0ce19a075348c5146b4\": rpc error: code = NotFound desc = could not find container \"dd02214c26380e2ef79dacdbf9badce4d57811fff29ca0ce19a075348c5146b4\": container with ID starting with dd02214c26380e2ef79dacdbf9badce4d57811fff29ca0ce19a075348c5146b4 not found: ID does not exist" Oct 31 13:53:23.028264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.028244 5532 scope.go:122] "RemoveContainer" containerID="bb552b3a689ad9b0102b281738bbfb59b6a006990466e803857fe92c6d429510" Oct 31 13:53:23.040202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.040180 5532 scope.go:122] "RemoveContainer" containerID="2735d3ea10d10be65c87f9c9ec36ca13c4c607e7d15a4bde348cdc030d896f9b" Oct 31 13:53:23.043195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.043124 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476"] Oct 31 13:53:23.048358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.048336 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-8b2f0a-openshift-default-64797b57bc-w9476"] Oct 31 13:53:23.051877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.051857 5532 scope.go:122] "RemoveContainer" containerID="bb552b3a689ad9b0102b281738bbfb59b6a006990466e803857fe92c6d429510" Oct 31 13:53:23.052158 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:53:23.052140 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb552b3a689ad9b0102b281738bbfb59b6a006990466e803857fe92c6d429510\": container with ID starting with bb552b3a689ad9b0102b281738bbfb59b6a006990466e803857fe92c6d429510 not found: ID does not exist" containerID="bb552b3a689ad9b0102b281738bbfb59b6a006990466e803857fe92c6d429510" Oct 31 13:53:23.052220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.052165 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb552b3a689ad9b0102b281738bbfb59b6a006990466e803857fe92c6d429510"} err="failed to get container status \"bb552b3a689ad9b0102b281738bbfb59b6a006990466e803857fe92c6d429510\": rpc error: code = NotFound desc = could not find container \"bb552b3a689ad9b0102b281738bbfb59b6a006990466e803857fe92c6d429510\": container with ID starting with bb552b3a689ad9b0102b281738bbfb59b6a006990466e803857fe92c6d429510 not found: ID does not exist" Oct 31 13:53:23.052220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.052181 5532 scope.go:122] "RemoveContainer" containerID="2735d3ea10d10be65c87f9c9ec36ca13c4c607e7d15a4bde348cdc030d896f9b" Oct 31 13:53:23.052443 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:53:23.052425 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2735d3ea10d10be65c87f9c9ec36ca13c4c607e7d15a4bde348cdc030d896f9b\": container with ID starting with 2735d3ea10d10be65c87f9c9ec36ca13c4c607e7d15a4bde348cdc030d896f9b not found: ID does not exist" containerID="2735d3ea10d10be65c87f9c9ec36ca13c4c607e7d15a4bde348cdc030d896f9b" Oct 31 13:53:23.052493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.052449 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2735d3ea10d10be65c87f9c9ec36ca13c4c607e7d15a4bde348cdc030d896f9b"} err="failed to get container status \"2735d3ea10d10be65c87f9c9ec36ca13c4c607e7d15a4bde348cdc030d896f9b\": rpc error: code = NotFound desc = could not find container \"2735d3ea10d10be65c87f9c9ec36ca13c4c607e7d15a4bde348cdc030d896f9b\": container with ID starting with 2735d3ea10d10be65c87f9c9ec36ca13c4c607e7d15a4bde348cdc030d896f9b not found: ID does not exist" Oct 31 13:53:23.062843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.062818 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq"] Oct 31 13:53:23.068771 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.068726 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-d85d82-openshift-default-7b77944d8f-ll7vq"] Oct 31 13:53:23.611143 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.611114 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782"] Oct 31 13:53:23.611508 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.611306 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" podUID="0a503272-9747-4a45-abdb-62cdbb0a2560" containerName="discovery" containerID="cri-o://8517e6c17953f75a28780bb81f5c23f37842b80fce97141470a51045e7f1b14f" gracePeriod=30 Oct 31 13:53:23.919237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:23.919207 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:53:24.002171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.002133 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a503272-9747-4a45-abdb-62cdbb0a2560-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a503272-9747-4a45-abdb-62cdbb0a2560-local-certs\") pod \"0a503272-9747-4a45-abdb-62cdbb0a2560\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " Oct 31 13:53:24.002354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.002205 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-kubeconfig\") pod \"0a503272-9747-4a45-abdb-62cdbb0a2560\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " Oct 31 13:53:24.002354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.002256 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-kube-api-access-g8xzq\" (UniqueName: \"kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-kube-api-access-g8xzq\") pod \"0a503272-9747-4a45-abdb-62cdbb0a2560\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " Oct 31 13:53:24.002443 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.002353 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-dns-cert\") pod \"0a503272-9747-4a45-abdb-62cdbb0a2560\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " Oct 31 13:53:24.002443 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.002381 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-token\" (UniqueName: \"kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-token\") pod \"0a503272-9747-4a45-abdb-62cdbb0a2560\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " Oct 31 13:53:24.002523 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.002450 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-ca-configmap\") pod \"0a503272-9747-4a45-abdb-62cdbb0a2560\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " Oct 31 13:53:24.002564 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.002524 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-cacerts\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-cacerts\") pod \"0a503272-9747-4a45-abdb-62cdbb0a2560\" (UID: \"0a503272-9747-4a45-abdb-62cdbb0a2560\") " Oct 31 13:53:24.006835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.006717 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a503272-9747-4a45-abdb-62cdbb0a2560-local-certs" pod "0a503272-9747-4a45-abdb-62cdbb0a2560" (UID: "0a503272-9747-4a45-abdb-62cdbb0a2560"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:24.007363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.007169 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-cacerts" pod "0a503272-9747-4a45-abdb-62cdbb0a2560" (UID: "0a503272-9747-4a45-abdb-62cdbb0a2560"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:53:24.007549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.007492 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-ca-configmap" pod "0a503272-9747-4a45-abdb-62cdbb0a2560" (UID: "0a503272-9747-4a45-abdb-62cdbb0a2560"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:24.009954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.009931 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" path="/var/lib/kubelet/pods/0ff1a789-0fbf-43ee-91dc-41eb7636c4e4/volumes" Oct 31 13:53:24.010635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.010610 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" path="/var/lib/kubelet/pods/8a7ef42d-5b18-414a-a0ff-492cc511c0db/volumes" Oct 31 13:53:24.011746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.011723 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-token" pod "0a503272-9747-4a45-abdb-62cdbb0a2560" (UID: "0a503272-9747-4a45-abdb-62cdbb0a2560"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:24.011857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.011756 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-kube-api-access-g8xzq" pod "0a503272-9747-4a45-abdb-62cdbb0a2560" (UID: "0a503272-9747-4a45-abdb-62cdbb0a2560"). InnerVolumeSpecName "kube-api-access-g8xzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:24.013358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.013321 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-kubeconfig" pod "0a503272-9747-4a45-abdb-62cdbb0a2560" (UID: "0a503272-9747-4a45-abdb-62cdbb0a2560"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:53:24.013632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.013612 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-dns-cert" pod "0a503272-9747-4a45-abdb-62cdbb0a2560" (UID: "0a503272-9747-4a45-abdb-62cdbb0a2560"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:53:24.014415 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.014390 5532 generic.go:356] "Generic (PLEG): container finished" podID="0a503272-9747-4a45-abdb-62cdbb0a2560" containerID="8517e6c17953f75a28780bb81f5c23f37842b80fce97141470a51045e7f1b14f" exitCode=0 Oct 31 13:53:24.014505 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.014489 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" Oct 31 13:53:24.040146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.040120 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" event={"ID":"0a503272-9747-4a45-abdb-62cdbb0a2560","Type":"ContainerDied","Data":"8517e6c17953f75a28780bb81f5c23f37842b80fce97141470a51045e7f1b14f"} Oct 31 13:53:24.040256 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.040157 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782" event={"ID":"0a503272-9747-4a45-abdb-62cdbb0a2560","Type":"ContainerDied","Data":"a995c9dd9fda9749ae79e3c5862bc3071b124ea2a069c30ba11e87c270d5f96a"} Oct 31 13:53:24.040256 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.040181 5532 scope.go:122] "RemoveContainer" containerID="8517e6c17953f75a28780bb81f5c23f37842b80fce97141470a51045e7f1b14f" Oct 31 13:53:24.052959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.052941 5532 scope.go:122] "RemoveContainer" containerID="8517e6c17953f75a28780bb81f5c23f37842b80fce97141470a51045e7f1b14f" Oct 31 13:53:24.053261 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:53:24.053240 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8517e6c17953f75a28780bb81f5c23f37842b80fce97141470a51045e7f1b14f\": container with ID starting with 8517e6c17953f75a28780bb81f5c23f37842b80fce97141470a51045e7f1b14f not found: ID does not exist" containerID="8517e6c17953f75a28780bb81f5c23f37842b80fce97141470a51045e7f1b14f" Oct 31 13:53:24.053321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.053270 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8517e6c17953f75a28780bb81f5c23f37842b80fce97141470a51045e7f1b14f"} err="failed to get container status \"8517e6c17953f75a28780bb81f5c23f37842b80fce97141470a51045e7f1b14f\": rpc error: code = NotFound desc = could not find container \"8517e6c17953f75a28780bb81f5c23f37842b80fce97141470a51045e7f1b14f\": container with ID starting with 8517e6c17953f75a28780bb81f5c23f37842b80fce97141470a51045e7f1b14f not found: ID does not exist" Oct 31 13:53:24.066191 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.066167 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782"] Oct 31 13:53:24.069959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.069941 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-bb782"] Oct 31 13:53:24.092512 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.092462 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" podUID="dd0271dc-44bc-41fc-9e77-a882c4fddd2e" containerName="payload-processing" containerID="cri-o://118ee0b5d079b2a2a0e4e5fde11c178bdc0d10746500ac042d9ce038fd4f9026" gracePeriod=25 Oct 31 13:53:24.105779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.105750 5532 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a503272-9747-4a45-abdb-62cdbb0a2560-local-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:24.105890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.105781 5532 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-kubeconfig\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:24.105890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.105799 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8xzq\" (UniqueName: \"kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-kube-api-access-g8xzq\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:24.105890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.105815 5532 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-dns-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:24.105890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.105829 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:24.105890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.105843 5532 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0a503272-9747-4a45-abdb-62cdbb0a2560-istio-csr-ca-configmap\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:24.105890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.105859 5532 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0a503272-9747-4a45-abdb-62cdbb0a2560-cacerts\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:24.107272 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.107242 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" podUID="1e663abe-09aa-4060-b7ad-dc235442ad97" containerName="payload-pre-processing" containerID="cri-o://9f58a5b7cf4af31d2a891e01d8db2037e972248e0a0edb6ff7f616a3fa380db2" gracePeriod=25 Oct 31 13:53:24.407045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.407019 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" Oct 31 13:53:24.428435 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.428408 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" Oct 31 13:53:24.512202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.512170 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-kube-api-access-hvxqx\" (UniqueName: \"kubernetes.io/projected/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-kube-api-access-hvxqx\") pod \"dd0271dc-44bc-41fc-9e77-a882c4fddd2e\" (UID: \"dd0271dc-44bc-41fc-9e77-a882c4fddd2e\") " Oct 31 13:53:24.512366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.512227 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-plugins-config-volume\") pod \"dd0271dc-44bc-41fc-9e77-a882c4fddd2e\" (UID: \"dd0271dc-44bc-41fc-9e77-a882c4fddd2e\") " Oct 31 13:53:24.512464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.512445 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e663abe-09aa-4060-b7ad-dc235442ad97-kube-api-access-wd8s4\" (UniqueName: \"kubernetes.io/projected/1e663abe-09aa-4060-b7ad-dc235442ad97-kube-api-access-wd8s4\") pod \"1e663abe-09aa-4060-b7ad-dc235442ad97\" (UID: \"1e663abe-09aa-4060-b7ad-dc235442ad97\") " Oct 31 13:53:24.512647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.512589 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e663abe-09aa-4060-b7ad-dc235442ad97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e663abe-09aa-4060-b7ad-dc235442ad97-plugins-config-volume\") pod \"1e663abe-09aa-4060-b7ad-dc235442ad97\" (UID: \"1e663abe-09aa-4060-b7ad-dc235442ad97\") " Oct 31 13:53:24.512741 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.512711 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-plugins-config-volume" pod "dd0271dc-44bc-41fc-9e77-a882c4fddd2e" (UID: "dd0271dc-44bc-41fc-9e77-a882c4fddd2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:24.513015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.512991 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e663abe-09aa-4060-b7ad-dc235442ad97-plugins-config-volume" pod "1e663abe-09aa-4060-b7ad-dc235442ad97" (UID: "1e663abe-09aa-4060-b7ad-dc235442ad97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:53:24.513317 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.513296 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:24.513373 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.513322 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e663abe-09aa-4060-b7ad-dc235442ad97-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:24.514384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.514365 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-kube-api-access-hvxqx" pod "dd0271dc-44bc-41fc-9e77-a882c4fddd2e" (UID: "dd0271dc-44bc-41fc-9e77-a882c4fddd2e"). InnerVolumeSpecName "kube-api-access-hvxqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:24.514528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.514508 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e663abe-09aa-4060-b7ad-dc235442ad97-kube-api-access-wd8s4" pod "1e663abe-09aa-4060-b7ad-dc235442ad97" (UID: "1e663abe-09aa-4060-b7ad-dc235442ad97"). InnerVolumeSpecName "kube-api-access-wd8s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:24.614303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.614264 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd8s4\" (UniqueName: \"kubernetes.io/projected/1e663abe-09aa-4060-b7ad-dc235442ad97-kube-api-access-wd8s4\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:24.614303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:24.614300 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvxqx\" (UniqueName: \"kubernetes.io/projected/dd0271dc-44bc-41fc-9e77-a882c4fddd2e-kube-api-access-hvxqx\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:25.032798 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.032757 5532 generic.go:356] "Generic (PLEG): container finished" podID="1e663abe-09aa-4060-b7ad-dc235442ad97" containerID="9f58a5b7cf4af31d2a891e01d8db2037e972248e0a0edb6ff7f616a3fa380db2" exitCode=0 Oct 31 13:53:25.033053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.032820 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" Oct 31 13:53:25.033053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.032911 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" event={"ID":"1e663abe-09aa-4060-b7ad-dc235442ad97","Type":"ContainerDied","Data":"9f58a5b7cf4af31d2a891e01d8db2037e972248e0a0edb6ff7f616a3fa380db2"} Oct 31 13:53:25.033053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.032962 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2" event={"ID":"1e663abe-09aa-4060-b7ad-dc235442ad97","Type":"ContainerDied","Data":"d57d95b56afedf92d2be039a8c6573d9c46ddd3caf2fa96e8edb311e54b2dcbe"} Oct 31 13:53:25.033053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.033013 5532 scope.go:122] "RemoveContainer" containerID="9f58a5b7cf4af31d2a891e01d8db2037e972248e0a0edb6ff7f616a3fa380db2" Oct 31 13:53:25.035777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.035725 5532 generic.go:356] "Generic (PLEG): container finished" podID="dd0271dc-44bc-41fc-9e77-a882c4fddd2e" containerID="118ee0b5d079b2a2a0e4e5fde11c178bdc0d10746500ac042d9ce038fd4f9026" exitCode=0 Oct 31 13:53:25.035856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.035810 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" event={"ID":"dd0271dc-44bc-41fc-9e77-a882c4fddd2e","Type":"ContainerDied","Data":"118ee0b5d079b2a2a0e4e5fde11c178bdc0d10746500ac042d9ce038fd4f9026"} Oct 31 13:53:25.035856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.035849 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" event={"ID":"dd0271dc-44bc-41fc-9e77-a882c4fddd2e","Type":"ContainerDied","Data":"796bcb9b80ab0a1d328078e33af8afa4dd00182d11ef9aade195b804f5867304"} Oct 31 13:53:25.035970 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.035922 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5" Oct 31 13:53:25.036890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.036866 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87"] Oct 31 13:53:25.047482 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.047416 5532 scope.go:122] "RemoveContainer" containerID="9f58a5b7cf4af31d2a891e01d8db2037e972248e0a0edb6ff7f616a3fa380db2" Oct 31 13:53:25.047743 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:53:25.047715 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f58a5b7cf4af31d2a891e01d8db2037e972248e0a0edb6ff7f616a3fa380db2\": container with ID starting with 9f58a5b7cf4af31d2a891e01d8db2037e972248e0a0edb6ff7f616a3fa380db2 not found: ID does not exist" containerID="9f58a5b7cf4af31d2a891e01d8db2037e972248e0a0edb6ff7f616a3fa380db2" Oct 31 13:53:25.047820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.047754 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f58a5b7cf4af31d2a891e01d8db2037e972248e0a0edb6ff7f616a3fa380db2"} err="failed to get container status \"9f58a5b7cf4af31d2a891e01d8db2037e972248e0a0edb6ff7f616a3fa380db2\": rpc error: code = NotFound desc = could not find container \"9f58a5b7cf4af31d2a891e01d8db2037e972248e0a0edb6ff7f616a3fa380db2\": container with ID starting with 9f58a5b7cf4af31d2a891e01d8db2037e972248e0a0edb6ff7f616a3fa380db2 not found: ID does not exist" Oct 31 13:53:25.047820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.047777 5532 scope.go:122] "RemoveContainer" containerID="118ee0b5d079b2a2a0e4e5fde11c178bdc0d10746500ac042d9ce038fd4f9026" Oct 31 13:53:25.059480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.059461 5532 scope.go:122] "RemoveContainer" containerID="118ee0b5d079b2a2a0e4e5fde11c178bdc0d10746500ac042d9ce038fd4f9026" Oct 31 13:53:25.059761 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:53:25.059739 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118ee0b5d079b2a2a0e4e5fde11c178bdc0d10746500ac042d9ce038fd4f9026\": container with ID starting with 118ee0b5d079b2a2a0e4e5fde11c178bdc0d10746500ac042d9ce038fd4f9026 not found: ID does not exist" containerID="118ee0b5d079b2a2a0e4e5fde11c178bdc0d10746500ac042d9ce038fd4f9026" Oct 31 13:53:25.059837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.059765 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118ee0b5d079b2a2a0e4e5fde11c178bdc0d10746500ac042d9ce038fd4f9026"} err="failed to get container status \"118ee0b5d079b2a2a0e4e5fde11c178bdc0d10746500ac042d9ce038fd4f9026\": rpc error: code = NotFound desc = could not find container \"118ee0b5d079b2a2a0e4e5fde11c178bdc0d10746500ac042d9ce038fd4f9026\": container with ID starting with 118ee0b5d079b2a2a0e4e5fde11c178bdc0d10746500ac042d9ce038fd4f9026 not found: ID does not exist" Oct 31 13:53:25.083046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.083024 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5"] Oct 31 13:53:25.087862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.087843 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-6dcb599f9f-hwmv5"] Oct 31 13:53:25.104035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.104014 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2"] Oct 31 13:53:25.107270 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:25.107250 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-6b7c898555-kb8f2"] Oct 31 13:53:26.004957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.004925 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a503272-9747-4a45-abdb-62cdbb0a2560" path="/var/lib/kubelet/pods/0a503272-9747-4a45-abdb-62cdbb0a2560/volumes" Oct 31 13:53:26.005399 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.005385 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e663abe-09aa-4060-b7ad-dc235442ad97" path="/var/lib/kubelet/pods/1e663abe-09aa-4060-b7ad-dc235442ad97/volumes" Oct 31 13:53:26.005796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.005782 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd0271dc-44bc-41fc-9e77-a882c4fddd2e" path="/var/lib/kubelet/pods/dd0271dc-44bc-41fc-9e77-a882c4fddd2e/volumes" Oct 31 13:53:26.296981 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.296907 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8"] Oct 31 13:53:26.297638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297623 5532 cpu_manager.go:414] "containerMap: removing container" podUID="5414af14-988d-4f31-9e65-479f5a6d07d7" containerName="revoke-keys" Oct 31 13:53:26.297712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297639 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5414af14-988d-4f31-9e65-479f5a6d07d7" containerName="revoke-keys" Oct 31 13:53:26.297712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297653 5532 cpu_manager.go:414] "containerMap: removing container" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerName="istio-proxy" Oct 31 13:53:26.297712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297658 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerName="istio-proxy" Oct 31 13:53:26.297712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297674 5532 cpu_manager.go:414] "containerMap: removing container" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerName="istio-proxy" Oct 31 13:53:26.297712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297679 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerName="istio-proxy" Oct 31 13:53:26.297712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297689 5532 cpu_manager.go:414] "containerMap: removing container" podUID="0a503272-9747-4a45-abdb-62cdbb0a2560" containerName="discovery" Oct 31 13:53:26.297712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297693 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a503272-9747-4a45-abdb-62cdbb0a2560" containerName="discovery" Oct 31 13:53:26.297712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297703 5532 cpu_manager.go:414] "containerMap: removing container" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerName="istio-proxy" Oct 31 13:53:26.297712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297708 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerName="istio-proxy" Oct 31 13:53:26.298030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297722 5532 cpu_manager.go:414] "containerMap: removing container" podUID="1e663abe-09aa-4060-b7ad-dc235442ad97" containerName="payload-pre-processing" Oct 31 13:53:26.298030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297726 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e663abe-09aa-4060-b7ad-dc235442ad97" containerName="payload-pre-processing" Oct 31 13:53:26.298030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297749 5532 cpu_manager.go:414] "containerMap: removing container" podUID="dd0271dc-44bc-41fc-9e77-a882c4fddd2e" containerName="payload-processing" Oct 31 13:53:26.298030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297754 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd0271dc-44bc-41fc-9e77-a882c4fddd2e" containerName="payload-processing" Oct 31 13:53:26.298030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297849 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e663abe-09aa-4060-b7ad-dc235442ad97" containerName="payload-pre-processing" Oct 31 13:53:26.298030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297858 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerName="istio-proxy" Oct 31 13:53:26.298030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297870 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd0271dc-44bc-41fc-9e77-a882c4fddd2e" containerName="payload-processing" Oct 31 13:53:26.298030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297880 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="5414af14-988d-4f31-9e65-479f5a6d07d7" containerName="revoke-keys" Oct 31 13:53:26.298030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297887 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerName="istio-proxy" Oct 31 13:53:26.298030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297894 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a503272-9747-4a45-abdb-62cdbb0a2560" containerName="discovery" Oct 31 13:53:26.298030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.297901 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ff1a789-0fbf-43ee-91dc-41eb7636c4e4" containerName="istio-proxy" Oct 31 13:53:26.303113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.303097 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.317428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.317403 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8"] Oct 31 13:53:26.359504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.359472 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4be675-openshift-default-dockercfg-5wxt2\"" Oct 31 13:53:26.441289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.441260 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-token\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.441481 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.441299 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-envoy\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.441481 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.441324 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-certs\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.441651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.441569 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-credential-socket\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.441651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.441633 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-socket\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.441777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.441761 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564lb\" (UniqueName: \"kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-kube-api-access-564lb\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.441837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.441799 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-podinfo\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.441929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.441909 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-data\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.441971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.441961 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4723d399-8244-4fd0-81f2-ce35a33441c8-istiod-ca-cert\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.546664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.544512 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-token\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.546664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.544581 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-envoy\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.546664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.544649 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-certs\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.546664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.544798 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-credential-socket\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.546664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.544840 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-socket\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.546664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.544927 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564lb\" (UniqueName: \"kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-kube-api-access-564lb\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.546664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.544974 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-podinfo\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.546664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.545038 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-data\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.546664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.545099 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4723d399-8244-4fd0-81f2-ce35a33441c8-istiod-ca-cert\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.546664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.546009 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4723d399-8244-4fd0-81f2-ce35a33441c8-istiod-ca-cert\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.547722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.547666 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-socket\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.548779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.548102 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-certs\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.548905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.548328 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-credential-socket\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.548991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.548936 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-data\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.590130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.589433 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-token\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.590130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.589993 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-564lb\" (UniqueName: \"kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-kube-api-access-564lb\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.591013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.590985 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-podinfo\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.617848 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-envoy\") pod \"e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.618919 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.618891 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:26.923482 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.923454 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8"] Oct 31 13:53:26.923651 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:53:26.923543 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4723d399_8244_4fd0_81f2_ce35a33441c8.slice/crio-da0969eeaea313275d64295061d3872f167b7f39f689fa80a497a698dacdfa42 WatchSource:0}: Error finding container da0969eeaea313275d64295061d3872f167b7f39f689fa80a497a698dacdfa42: Status 404 returned error can't find the container with id da0969eeaea313275d64295061d3872f167b7f39f689fa80a497a698dacdfa42 Oct 31 13:53:26.926542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.926508 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:53:26.926657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.926587 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:53:26.926657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:26.926649 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:53:27.063304 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:27.063243 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" event={"ID":"4723d399-8244-4fd0-81f2-ce35a33441c8","Type":"ContainerStarted","Data":"c79c50c5d0949535c4d1acb318fdcc07c4b14bfa3d7478b34271be8a1112662e"} Oct 31 13:53:27.063836 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:27.063368 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" event={"ID":"4723d399-8244-4fd0-81f2-ce35a33441c8","Type":"ContainerStarted","Data":"da0969eeaea313275d64295061d3872f167b7f39f689fa80a497a698dacdfa42"} Oct 31 13:53:27.093422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:27.093358 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" podStartSLOduration=1.0933417109999999 podStartE2EDuration="1.093341711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:27.092976475 +0000 UTC m=+2975.238720307" watchObservedRunningTime="2026-10-31 13:53:27.093341711 +0000 UTC m=+2975.239085542" Oct 31 13:53:27.620608 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:27.620569 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:28.627978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:28.627949 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:29.079948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:29.079917 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:29.080800 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:29.080780 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:53:30.480660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.480557 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz"] Oct 31 13:53:30.481386 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.481368 5532 cpu_manager.go:414] "containerMap: removing container" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerName="istio-proxy" Oct 31 13:53:30.481430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.481388 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerName="istio-proxy" Oct 31 13:53:30.481528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.481518 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7ef42d-5b18-414a-a0ff-492cc511c0db" containerName="istio-proxy" Oct 31 13:53:30.485711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.485686 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.500018 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.499986 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz"] Oct 31 13:53:30.560116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.560079 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-a81979-openshift-default-dockercfg-bg46j\"" Oct 31 13:53:30.659617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.640176 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.659929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.659901 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.660243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.660220 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.660364 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.660347 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.660679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.660656 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgskr\" (UniqueName: \"kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-kube-api-access-xgskr\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.678621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.660837 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.678954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.678917 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.679166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.679148 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee59769-d44a-4690-b5b0-aeecb07beff2-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.679298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.679283 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.781167 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.781087 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgskr\" (UniqueName: \"kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-kube-api-access-xgskr\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.781167 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.781139 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.781407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.781304 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.781407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.781353 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee59769-d44a-4690-b5b0-aeecb07beff2-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.781407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.781375 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.781562 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.781489 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.781562 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.781496 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.781758 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.781575 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.781831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.781803 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.782133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.782109 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee59769-d44a-4690-b5b0-aeecb07beff2-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.782227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.782177 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.782227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.782200 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.782462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.782439 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.782588 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.782558 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.783946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.783926 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.784910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.784884 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.799400 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.799372 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.801083 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.801045 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgskr\" (UniqueName: \"kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-kube-api-access-xgskr\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.809393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.809369 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:30.949160 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.949122 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4"] Oct 31 13:53:30.955913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.955884 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" Oct 31 13:53:30.956601 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.956547 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt"] Oct 31 13:53:30.965105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.964111 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4be675-dockercfg-jtsm6\"" Oct 31 13:53:30.965105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.964203 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4be675\"" Oct 31 13:53:30.972536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.972513 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4"] Oct 31 13:53:30.972871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.972809 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" Oct 31 13:53:30.977130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:30.976857 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt"] Oct 31 13:53:31.095060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.094952 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86524ce1-913b-4bc9-97b8-6adb65d9f702-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt\" (UID: \"86524ce1-913b-4bc9-97b8-6adb65d9f702\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" Oct 31 13:53:31.095332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.095300 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ea32b8-16bd-4ada-a51f-ba0e638a1457-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4\" (UID: \"46ea32b8-16bd-4ada-a51f-ba0e638a1457\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" Oct 31 13:53:31.095449 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.095427 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d58b8\" (UniqueName: \"kubernetes.io/projected/46ea32b8-16bd-4ada-a51f-ba0e638a1457-kube-api-access-d58b8\") pod \"payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4\" (UID: \"46ea32b8-16bd-4ada-a51f-ba0e638a1457\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" Oct 31 13:53:31.095527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.095482 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rjvw\" (UniqueName: \"kubernetes.io/projected/86524ce1-913b-4bc9-97b8-6adb65d9f702-kube-api-access-6rjvw\") pod \"payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt\" (UID: \"86524ce1-913b-4bc9-97b8-6adb65d9f702\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" Oct 31 13:53:31.172199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.172159 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:53:31.172342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.172233 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:53:31.172342 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.172262 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:53:31.182765 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.182501 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz"] Oct 31 13:53:31.221642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.221581 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86524ce1-913b-4bc9-97b8-6adb65d9f702-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt\" (UID: \"86524ce1-913b-4bc9-97b8-6adb65d9f702\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" Oct 31 13:53:31.221840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.221774 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ea32b8-16bd-4ada-a51f-ba0e638a1457-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4\" (UID: \"46ea32b8-16bd-4ada-a51f-ba0e638a1457\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" Oct 31 13:53:31.221840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.221838 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d58b8\" (UniqueName: \"kubernetes.io/projected/46ea32b8-16bd-4ada-a51f-ba0e638a1457-kube-api-access-d58b8\") pod \"payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4\" (UID: \"46ea32b8-16bd-4ada-a51f-ba0e638a1457\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" Oct 31 13:53:31.221982 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.221868 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rjvw\" (UniqueName: \"kubernetes.io/projected/86524ce1-913b-4bc9-97b8-6adb65d9f702-kube-api-access-6rjvw\") pod \"payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt\" (UID: \"86524ce1-913b-4bc9-97b8-6adb65d9f702\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" Oct 31 13:53:31.222484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.222463 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86524ce1-913b-4bc9-97b8-6adb65d9f702-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt\" (UID: \"86524ce1-913b-4bc9-97b8-6adb65d9f702\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" Oct 31 13:53:31.222647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.222626 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ea32b8-16bd-4ada-a51f-ba0e638a1457-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4\" (UID: \"46ea32b8-16bd-4ada-a51f-ba0e638a1457\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" Oct 31 13:53:31.236335 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.236310 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rjvw\" (UniqueName: \"kubernetes.io/projected/86524ce1-913b-4bc9-97b8-6adb65d9f702-kube-api-access-6rjvw\") pod \"payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt\" (UID: \"86524ce1-913b-4bc9-97b8-6adb65d9f702\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" Oct 31 13:53:31.236465 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.236443 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d58b8\" (UniqueName: \"kubernetes.io/projected/46ea32b8-16bd-4ada-a51f-ba0e638a1457-kube-api-access-d58b8\") pod \"payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4\" (UID: \"46ea32b8-16bd-4ada-a51f-ba0e638a1457\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" Oct 31 13:53:31.293356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.293325 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" Oct 31 13:53:31.304916 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.304882 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" Oct 31 13:53:31.603832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.603803 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4"] Oct 31 13:53:31.604240 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:53:31.604118 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46ea32b8_16bd_4ada_a51f_ba0e638a1457.slice/crio-a1496f7b8951c8cfe0baeb873365ace80b2fb4133c3d9e049b5107cfce08ac49 WatchSource:0}: Error finding container a1496f7b8951c8cfe0baeb873365ace80b2fb4133c3d9e049b5107cfce08ac49: Status 404 returned error can't find the container with id a1496f7b8951c8cfe0baeb873365ace80b2fb4133c3d9e049b5107cfce08ac49 Oct 31 13:53:31.612094 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:31.612052 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt"] Oct 31 13:53:31.612396 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:53:31.612372 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86524ce1_913b_4bc9_97b8_6adb65d9f702.slice/crio-75dfc7e08c823a43182d359fd1111c9affe57d21925d220b349e00f7e03645ad WatchSource:0}: Error finding container 75dfc7e08c823a43182d359fd1111c9affe57d21925d220b349e00f7e03645ad: Status 404 returned error can't find the container with id 75dfc7e08c823a43182d359fd1111c9affe57d21925d220b349e00f7e03645ad Oct 31 13:53:31.765130 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:53:31.765092 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:53:32.183770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:32.183734 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" event={"ID":"86524ce1-913b-4bc9-97b8-6adb65d9f702","Type":"ContainerStarted","Data":"e01c9a4e238e20db457860e965e2e991d0ed94bb04031ffaed3bb369cc959964"} Oct 31 13:53:32.183968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:32.183799 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" Oct 31 13:53:32.183968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:32.183822 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" event={"ID":"86524ce1-913b-4bc9-97b8-6adb65d9f702","Type":"ContainerStarted","Data":"75dfc7e08c823a43182d359fd1111c9affe57d21925d220b349e00f7e03645ad"} Oct 31 13:53:32.185409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:32.185378 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" event={"ID":"7ee59769-d44a-4690-b5b0-aeecb07beff2","Type":"ContainerStarted","Data":"ba38cb398433bb8c326aaf9d913b4c09baac5b5cef2b5f9c06ff230c752c872e"} Oct 31 13:53:32.185548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:32.185427 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" event={"ID":"7ee59769-d44a-4690-b5b0-aeecb07beff2","Type":"ContainerStarted","Data":"5cfe696e7e671598042148c9aa7d5fe4f7fc9dd02a986ed49327b6dd9a7c4e36"} Oct 31 13:53:32.187138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:32.187111 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" event={"ID":"46ea32b8-16bd-4ada-a51f-ba0e638a1457","Type":"ContainerStarted","Data":"75d076416d262d6e46f412e2c53d5005ebe6e37f77c2f9b5d118719dee5cb5b7"} Oct 31 13:53:32.187255 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:32.187168 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" event={"ID":"46ea32b8-16bd-4ada-a51f-ba0e638a1457","Type":"ContainerStarted","Data":"a1496f7b8951c8cfe0baeb873365ace80b2fb4133c3d9e049b5107cfce08ac49"} Oct 31 13:53:32.187255 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:32.187198 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" Oct 31 13:53:32.209056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:32.209008 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" podStartSLOduration=2.208990044 podStartE2EDuration="2.208990044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:32.208013446 +0000 UTC m=+2980.353757275" watchObservedRunningTime="2026-10-31 13:53:32.208990044 +0000 UTC m=+2980.354733873" Oct 31 13:53:32.240363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:32.240312 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" podStartSLOduration=2.240299441 podStartE2EDuration="2.240299441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:32.23762163 +0000 UTC m=+2980.383365464" watchObservedRunningTime="2026-10-31 13:53:32.240299441 +0000 UTC m=+2980.386043268" Oct 31 13:53:32.260993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:32.260948 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" podStartSLOduration=2.260935116 podStartE2EDuration="2.260935116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:32.256583065 +0000 UTC m=+2980.402326895" watchObservedRunningTime="2026-10-31 13:53:32.260935116 +0000 UTC m=+2980.406678944" Oct 31 13:53:32.809852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:32.809819 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:32.816456 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:32.816429 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:33.195709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:33.195673 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:33.196686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:33.196665 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:53:35.950737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:35.950704 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx"] Oct 31 13:53:35.956411 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:35.956380 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" Oct 31 13:53:35.968761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:35.968729 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx"] Oct 31 13:53:35.969738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:35.969709 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a81979\"" Oct 31 13:53:35.970796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:35.970766 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a81979-dockercfg-lktd8\"" Oct 31 13:53:35.977551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:35.977523 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f"] Oct 31 13:53:35.983315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:35.983295 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" Oct 31 13:53:36.012236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.012206 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f"] Oct 31 13:53:36.050033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.050000 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9356e238-adec-4f2e-bdd0-e9e369a660df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx\" (UID: \"9356e238-adec-4f2e-bdd0-e9e369a660df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" Oct 31 13:53:36.050213 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.050196 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rbm7\" (UniqueName: \"kubernetes.io/projected/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-kube-api-access-7rbm7\") pod \"payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f\" (UID: \"c3b4c605-cffc-4cfa-aed6-20d22c222fa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" Oct 31 13:53:36.050437 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.050408 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f\" (UID: \"c3b4c605-cffc-4cfa-aed6-20d22c222fa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" Oct 31 13:53:36.050513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.050458 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rjr\" (UniqueName: \"kubernetes.io/projected/9356e238-adec-4f2e-bdd0-e9e369a660df-kube-api-access-64rjr\") pod \"payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx\" (UID: \"9356e238-adec-4f2e-bdd0-e9e369a660df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" Oct 31 13:53:36.163617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.162839 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rbm7\" (UniqueName: \"kubernetes.io/projected/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-kube-api-access-7rbm7\") pod \"payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f\" (UID: \"c3b4c605-cffc-4cfa-aed6-20d22c222fa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" Oct 31 13:53:36.163617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.163053 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f\" (UID: \"c3b4c605-cffc-4cfa-aed6-20d22c222fa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" Oct 31 13:53:36.163617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.163104 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rjr\" (UniqueName: \"kubernetes.io/projected/9356e238-adec-4f2e-bdd0-e9e369a660df-kube-api-access-64rjr\") pod \"payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx\" (UID: \"9356e238-adec-4f2e-bdd0-e9e369a660df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" Oct 31 13:53:36.163617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.163149 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9356e238-adec-4f2e-bdd0-e9e369a660df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx\" (UID: \"9356e238-adec-4f2e-bdd0-e9e369a660df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" Oct 31 13:53:36.164747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.164161 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9356e238-adec-4f2e-bdd0-e9e369a660df-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx\" (UID: \"9356e238-adec-4f2e-bdd0-e9e369a660df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" Oct 31 13:53:36.179618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.165253 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f\" (UID: \"c3b4c605-cffc-4cfa-aed6-20d22c222fa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" Oct 31 13:53:36.199618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.195264 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rbm7\" (UniqueName: \"kubernetes.io/projected/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-kube-api-access-7rbm7\") pod \"payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f\" (UID: \"c3b4c605-cffc-4cfa-aed6-20d22c222fa2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" Oct 31 13:53:36.199618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.196421 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rjr\" (UniqueName: \"kubernetes.io/projected/9356e238-adec-4f2e-bdd0-e9e369a660df-kube-api-access-64rjr\") pod \"payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx\" (UID: \"9356e238-adec-4f2e-bdd0-e9e369a660df\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" Oct 31 13:53:36.275002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.274918 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" Oct 31 13:53:36.308559 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.308529 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" Oct 31 13:53:36.566910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.566880 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx"] Oct 31 13:53:36.567196 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:53:36.567172 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9356e238_adec_4f2e_bdd0_e9e369a660df.slice/crio-13493b1a0ea27d13e7eb8dea832b1d1cf2be77624ef942481fe2a3f46e5e3f65 WatchSource:0}: Error finding container 13493b1a0ea27d13e7eb8dea832b1d1cf2be77624ef942481fe2a3f46e5e3f65: Status 404 returned error can't find the container with id 13493b1a0ea27d13e7eb8dea832b1d1cf2be77624ef942481fe2a3f46e5e3f65 Oct 31 13:53:36.614904 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:53:36.614854 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3b4c605_cffc_4cfa_aed6_20d22c222fa2.slice/crio-ed9847fe1a0781bf31e8e5c388708b4572c237f49648fabfe8ed4050fa7c4ae2 WatchSource:0}: Error finding container ed9847fe1a0781bf31e8e5c388708b4572c237f49648fabfe8ed4050fa7c4ae2: Status 404 returned error can't find the container with id ed9847fe1a0781bf31e8e5c388708b4572c237f49648fabfe8ed4050fa7c4ae2 Oct 31 13:53:36.615042 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:36.614910 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f"] Oct 31 13:53:37.231867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:37.231827 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" event={"ID":"9356e238-adec-4f2e-bdd0-e9e369a660df","Type":"ContainerStarted","Data":"40e0750560b1316c00b792ce360dd9acaab1dc2ea1d078acb8f99380f27891ab"} Oct 31 13:53:37.232357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:37.231896 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" event={"ID":"9356e238-adec-4f2e-bdd0-e9e369a660df","Type":"ContainerStarted","Data":"13493b1a0ea27d13e7eb8dea832b1d1cf2be77624ef942481fe2a3f46e5e3f65"} Oct 31 13:53:37.232357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:37.231925 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" Oct 31 13:53:37.233874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:37.233836 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" event={"ID":"c3b4c605-cffc-4cfa-aed6-20d22c222fa2","Type":"ContainerStarted","Data":"6108f22d8c27bdc6f9a9330f40ffd8aba5b4e1c18eb020485bc249900421469b"} Oct 31 13:53:37.234012 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:37.233902 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" Oct 31 13:53:37.234012 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:37.233929 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" event={"ID":"c3b4c605-cffc-4cfa-aed6-20d22c222fa2","Type":"ContainerStarted","Data":"ed9847fe1a0781bf31e8e5c388708b4572c237f49648fabfe8ed4050fa7c4ae2"} Oct 31 13:53:37.252915 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:37.252860 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" podStartSLOduration=2.2528434219999998 podStartE2EDuration="2.252843422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:37.251904555 +0000 UTC m=+2985.397648387" watchObservedRunningTime="2026-10-31 13:53:37.252843422 +0000 UTC m=+2985.398587251" Oct 31 13:53:37.275813 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:37.275770 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" podStartSLOduration=2.275756753 podStartE2EDuration="2.275756753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:37.274662571 +0000 UTC m=+2985.420406401" watchObservedRunningTime="2026-10-31 13:53:37.275756753 +0000 UTC m=+2985.421500630" Oct 31 13:53:40.062229 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.062188 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" podUID="8e487890-211d-4588-90c6-52499fc28ce0" containerName="main" containerID="cri-o://195a9e1017bc68407880d033a43715911179e4d9cf3d7637eae56f8af265dc89" gracePeriod=45 Oct 31 13:53:40.240382 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.240343 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" podUID="8e487890-211d-4588-90c6-52499fc28ce0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.216:8000/ready\": dial tcp 10.132.0.216:8000: connect: connection refused" Oct 31 13:53:40.279912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.279735 5532 generic.go:356] "Generic (PLEG): container finished" podID="8e487890-211d-4588-90c6-52499fc28ce0" containerID="195a9e1017bc68407880d033a43715911179e4d9cf3d7637eae56f8af265dc89" exitCode=0 Oct 31 13:53:40.279912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.279846 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" event={"ID":"8e487890-211d-4588-90c6-52499fc28ce0","Type":"ContainerDied","Data":"195a9e1017bc68407880d033a43715911179e4d9cf3d7637eae56f8af265dc89"} Oct 31 13:53:40.358109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.358079 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:53:40.430705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.430670 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-dshm\") pod \"8e487890-211d-4588-90c6-52499fc28ce0\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " Oct 31 13:53:40.430871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.430767 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-home\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-home\") pod \"8e487890-211d-4588-90c6-52499fc28ce0\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " Oct 31 13:53:40.430871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.430807 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-model-cache\") pod \"8e487890-211d-4588-90c6-52499fc28ce0\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " Oct 31 13:53:40.430967 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.430950 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e487890-211d-4588-90c6-52499fc28ce0-kube-api-access-2xzld\" (UniqueName: \"kubernetes.io/projected/8e487890-211d-4588-90c6-52499fc28ce0-kube-api-access-2xzld\") pod \"8e487890-211d-4588-90c6-52499fc28ce0\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " Oct 31 13:53:40.431038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.431011 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-home" pod "8e487890-211d-4588-90c6-52499fc28ce0" (UID: "8e487890-211d-4588-90c6-52499fc28ce0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:40.431136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.431110 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-model-cache" pod "8e487890-211d-4588-90c6-52499fc28ce0" (UID: "8e487890-211d-4588-90c6-52499fc28ce0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:40.431136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.431116 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-tmp-dir\") pod \"8e487890-211d-4588-90c6-52499fc28ce0\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " Oct 31 13:53:40.431261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.431248 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-tmp-dir" pod "8e487890-211d-4588-90c6-52499fc28ce0" (UID: "8e487890-211d-4588-90c6-52499fc28ce0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:40.431321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.431267 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e487890-211d-4588-90c6-52499fc28ce0-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e487890-211d-4588-90c6-52499fc28ce0-tls-certs\") pod \"8e487890-211d-4588-90c6-52499fc28ce0\" (UID: \"8e487890-211d-4588-90c6-52499fc28ce0\") " Oct 31 13:53:40.431828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.431810 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:40.431828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.431827 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:40.432007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.431836 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:40.433069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.433049 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-dshm" pod "8e487890-211d-4588-90c6-52499fc28ce0" (UID: "8e487890-211d-4588-90c6-52499fc28ce0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:53:40.433208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.433185 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e487890-211d-4588-90c6-52499fc28ce0-tls-certs" pod "8e487890-211d-4588-90c6-52499fc28ce0" (UID: "8e487890-211d-4588-90c6-52499fc28ce0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:53:40.433252 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.433241 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e487890-211d-4588-90c6-52499fc28ce0-kube-api-access-2xzld" pod "8e487890-211d-4588-90c6-52499fc28ce0" (UID: "8e487890-211d-4588-90c6-52499fc28ce0"). InnerVolumeSpecName "kube-api-access-2xzld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:53:40.533178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.533140 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e487890-211d-4588-90c6-52499fc28ce0-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:40.533178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.533171 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8e487890-211d-4588-90c6-52499fc28ce0-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:40.533178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.533183 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xzld\" (UniqueName: \"kubernetes.io/projected/8e487890-211d-4588-90c6-52499fc28ce0-kube-api-access-2xzld\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:53:40.928365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.928328 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w"] Oct 31 13:53:40.929121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.929104 5532 cpu_manager.go:414] "containerMap: removing container" podUID="8e487890-211d-4588-90c6-52499fc28ce0" containerName="main" Oct 31 13:53:40.929203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.929122 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e487890-211d-4588-90c6-52499fc28ce0" containerName="main" Oct 31 13:53:40.929308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.929298 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e487890-211d-4588-90c6-52499fc28ce0" containerName="main" Oct 31 13:53:40.933303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.933283 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" Oct 31 13:53:40.942156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.942130 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-20b8a4\"" Oct 31 13:53:40.943249 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.943214 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w"] Oct 31 13:53:40.947439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.947417 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk"] Oct 31 13:53:40.952709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.952685 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" Oct 31 13:53:40.968770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.968741 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk"] Oct 31 13:53:40.969638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:40.969616 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-20b8a4-dockercfg-sh4wd\"" Oct 31 13:53:41.041824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.041790 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7981c832-1940-492c-a03a-96dd7a8d473e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w\" (UID: \"7981c832-1940-492c-a03a-96dd7a8d473e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" Oct 31 13:53:41.042012 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.041988 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk\" (UID: \"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" Oct 31 13:53:41.042060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.042046 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26txg\" (UniqueName: \"kubernetes.io/projected/7981c832-1940-492c-a03a-96dd7a8d473e-kube-api-access-26txg\") pod \"payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w\" (UID: \"7981c832-1940-492c-a03a-96dd7a8d473e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" Oct 31 13:53:41.042101 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.042087 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69qxq\" (UniqueName: \"kubernetes.io/projected/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-kube-api-access-69qxq\") pod \"payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk\" (UID: \"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" Oct 31 13:53:41.144202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.144152 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk\" (UID: \"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" Oct 31 13:53:41.144740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.144713 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26txg\" (UniqueName: \"kubernetes.io/projected/7981c832-1940-492c-a03a-96dd7a8d473e-kube-api-access-26txg\") pod \"payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w\" (UID: \"7981c832-1940-492c-a03a-96dd7a8d473e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" Oct 31 13:53:41.144850 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.144754 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69qxq\" (UniqueName: \"kubernetes.io/projected/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-kube-api-access-69qxq\") pod \"payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk\" (UID: \"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" Oct 31 13:53:41.144956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.144938 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7981c832-1940-492c-a03a-96dd7a8d473e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w\" (UID: \"7981c832-1940-492c-a03a-96dd7a8d473e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" Oct 31 13:53:41.145777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.145749 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7981c832-1940-492c-a03a-96dd7a8d473e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w\" (UID: \"7981c832-1940-492c-a03a-96dd7a8d473e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" Oct 31 13:53:41.146653 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.146588 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk\" (UID: \"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" Oct 31 13:53:41.162446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.162418 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69qxq\" (UniqueName: \"kubernetes.io/projected/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-kube-api-access-69qxq\") pod \"payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk\" (UID: \"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" Oct 31 13:53:41.212167 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.209297 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26txg\" (UniqueName: \"kubernetes.io/projected/7981c832-1940-492c-a03a-96dd7a8d473e-kube-api-access-26txg\") pod \"payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w\" (UID: \"7981c832-1940-492c-a03a-96dd7a8d473e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" Oct 31 13:53:41.252812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.252778 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" Oct 31 13:53:41.282421 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.282392 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" Oct 31 13:53:41.293403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.293366 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" event={"ID":"8e487890-211d-4588-90c6-52499fc28ce0","Type":"ContainerDied","Data":"2de2043f3b3e3b18f1fb838c0715798380d5369c36222747c25e6d3d23cee76b"} Oct 31 13:53:41.293403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.293396 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87" Oct 31 13:53:41.293669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.293433 5532 scope.go:122] "RemoveContainer" containerID="195a9e1017bc68407880d033a43715911179e4d9cf3d7637eae56f8af265dc89" Oct 31 13:53:41.348327 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.346895 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87"] Oct 31 13:53:41.360635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.359975 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-612b46/auth-test-model-612b46-kserve-6f94769ddf-gvc87"] Oct 31 13:53:41.547609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.547561 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w"] Oct 31 13:53:41.548344 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:53:41.548321 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7981c832_1940_492c_a03a_96dd7a8d473e.slice/crio-5f2ad9767a3dcf61790717cb9ef400daaf83e9a995a44d81c56d62ddb23b42cf WatchSource:0}: Error finding container 5f2ad9767a3dcf61790717cb9ef400daaf83e9a995a44d81c56d62ddb23b42cf: Status 404 returned error can't find the container with id 5f2ad9767a3dcf61790717cb9ef400daaf83e9a995a44d81c56d62ddb23b42cf Oct 31 13:53:41.550306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.550290 5532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 13:53:41.600796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:41.600768 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk"] Oct 31 13:53:41.603157 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:53:41.603121 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eef8cf1_7c9e_489c_8ec6_0b5175e28e01.slice/crio-3a45f834cf4901a809962a79391a2fbed95effbbc25efee0d581916d3ac2a806 WatchSource:0}: Error finding container 3a45f834cf4901a809962a79391a2fbed95effbbc25efee0d581916d3ac2a806: Status 404 returned error can't find the container with id 3a45f834cf4901a809962a79391a2fbed95effbbc25efee0d581916d3ac2a806 Oct 31 13:53:42.016515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:42.016476 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e487890-211d-4588-90c6-52499fc28ce0" path="/var/lib/kubelet/pods/8e487890-211d-4588-90c6-52499fc28ce0/volumes" Oct 31 13:53:42.310897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:42.310761 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" event={"ID":"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01","Type":"ContainerStarted","Data":"3a45f834cf4901a809962a79391a2fbed95effbbc25efee0d581916d3ac2a806"} Oct 31 13:53:42.316385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:42.316347 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" event={"ID":"7981c832-1940-492c-a03a-96dd7a8d473e","Type":"ContainerStarted","Data":"5f2ad9767a3dcf61790717cb9ef400daaf83e9a995a44d81c56d62ddb23b42cf"} Oct 31 13:53:43.199030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:43.198999 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" Oct 31 13:53:43.199399 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:43.199371 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" Oct 31 13:53:44.339423 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:44.339389 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" event={"ID":"7981c832-1940-492c-a03a-96dd7a8d473e","Type":"ContainerStarted","Data":"b9475fd85cd13f152275e658d5d399c779ab1f3e35f30dcaaaf21b6d28c718bb"} Oct 31 13:53:44.340055 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:44.340031 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" Oct 31 13:53:44.341363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:44.341327 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" event={"ID":"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01","Type":"ContainerStarted","Data":"9f046bd77be5df61c394c55e4747905839ecc799100720dbe8c5fe1eafddb9e9"} Oct 31 13:53:44.341476 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:44.341397 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" Oct 31 13:53:44.364685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:44.364633 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" podStartSLOduration=2.2926756409999998 podStartE2EDuration="4.364616549s" totalImagesPullingTime="2.071940908s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:44.361626299 +0000 UTC m=+2992.507370128" watchObservedRunningTime="2026-10-31 13:53:44.364616549 +0000 UTC m=+2992.510360375" Oct 31 13:53:44.387555 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:44.387502 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" podStartSLOduration=2.385030112 podStartE2EDuration="4.387488694s" totalImagesPullingTime="2.002458582s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:44.384730552 +0000 UTC m=+2992.530474381" watchObservedRunningTime="2026-10-31 13:53:44.387488694 +0000 UTC m=+2992.533232555" Oct 31 13:53:48.245189 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:48.245157 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" Oct 31 13:53:48.245723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:48.245242 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" Oct 31 13:53:54.009138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:54.009106 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:53:55.351890 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:55.351857 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" Oct 31 13:53:55.352537 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:55.352019 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" Oct 31 13:53:57.009484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.008118 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn"] Oct 31 13:53:57.019628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.017877 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.020182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.020151 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn"] Oct 31 13:53:57.023377 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.023346 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8a47d6-openshift-default-dockercfg-ss75h\"" Oct 31 13:53:57.135007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.134972 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-certs\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.135207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.135027 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-data\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.135207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.135142 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-podinfo\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.135294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.135214 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-token\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.135347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.135327 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgckm\" (UniqueName: \"kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-kube-api-access-rgckm\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.135401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.135387 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-envoy\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.135438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.135418 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-credential-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.135670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.135649 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istiod-ca-cert\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.135775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.135763 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.237051 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.236975 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-certs\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.237231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.237081 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-data\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.237231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.237125 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-podinfo\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.237231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.237186 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-token\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.237434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.237232 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgckm\" (UniqueName: \"kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-kube-api-access-rgckm\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.237434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.237267 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-envoy\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.237434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.237295 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-credential-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.237434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.237382 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istiod-ca-cert\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.237686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.237431 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-certs\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.237686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.237499 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.237805 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.237785 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-credential-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.237950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.237926 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-data\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.238123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.238095 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-socket\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.238268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.238251 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istiod-ca-cert\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.239997 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.239964 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-envoy\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.240264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.240244 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-podinfo\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.252139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.252115 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgckm\" (UniqueName: \"kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-kube-api-access-rgckm\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.270014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.269953 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-token\") pod \"e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.339369 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.339334 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:57.642113 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:53:57.642074 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccbf2fa_9e8d_453b_ba9d_60dae20a4884.slice/crio-4274105f86bdf17174661ba6d34900a9cb9790910927e1840e0ba671e238b876 WatchSource:0}: Error finding container 4274105f86bdf17174661ba6d34900a9cb9790910927e1840e0ba671e238b876: Status 404 returned error can't find the container with id 4274105f86bdf17174661ba6d34900a9cb9790910927e1840e0ba671e238b876 Oct 31 13:53:57.642390 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.642353 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn"] Oct 31 13:53:57.646407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.646367 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:53:57.646528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.646445 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:53:57.646528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:57.646477 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:53:58.524978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:58.524932 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" event={"ID":"dccbf2fa-9e8d-453b-ba9d-60dae20a4884","Type":"ContainerStarted","Data":"92ffbb7a5f36d38e286f158c59ff493e16dc49698bfa76fe7ad991a1a7167a65"} Oct 31 13:53:58.525411 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:58.525001 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" event={"ID":"dccbf2fa-9e8d-453b-ba9d-60dae20a4884","Type":"ContainerStarted","Data":"4274105f86bdf17174661ba6d34900a9cb9790910927e1840e0ba671e238b876"} Oct 31 13:53:58.556517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:58.556451 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" podStartSLOduration=2.556431396 podStartE2EDuration="2.556431396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:53:58.554073529 +0000 UTC m=+3006.699817358" watchObservedRunningTime="2026-10-31 13:53:58.556431396 +0000 UTC m=+3006.702175225" Oct 31 13:53:59.339667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:59.339630 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:59.346008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:59.345981 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:59.537061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:59.537027 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:53:59.537977 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:53:59.537951 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:54:01.763683 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:54:01.763646 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:54:15.046155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:15.046121 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4"] Oct 31 13:54:15.061543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:15.061516 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt"] Oct 31 13:54:16.027586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:16.027554 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x"] Oct 31 13:54:16.033151 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:16.033124 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0f1b7cb3-tmz2x"] Oct 31 13:54:18.005442 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:18.005409 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8802788c-b651-4dbf-9281-98620e7946ae" path="/var/lib/kubelet/pods/8802788c-b651-4dbf-9281-98620e7946ae/volumes" Oct 31 13:54:20.046666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.046626 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" podUID="46ea32b8-16bd-4ada-a51f-ba0e638a1457" containerName="payload-processing" containerID="cri-o://75d076416d262d6e46f412e2c53d5005ebe6e37f77c2f9b5d118719dee5cb5b7" gracePeriod=25 Oct 31 13:54:20.062236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.062191 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" podUID="86524ce1-913b-4bc9-97b8-6adb65d9f702" containerName="payload-pre-processing" containerID="cri-o://e01c9a4e238e20db457860e965e2e991d0ed94bb04031ffaed3bb369cc959964" gracePeriod=25 Oct 31 13:54:20.221674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.221637 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f"] Oct 31 13:54:20.238736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.238703 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx"] Oct 31 13:54:20.407942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.407378 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" Oct 31 13:54:20.439835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.439805 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" Oct 31 13:54:20.461754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.461716 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46ea32b8-16bd-4ada-a51f-ba0e638a1457-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ea32b8-16bd-4ada-a51f-ba0e638a1457-plugins-config-volume\") pod \"46ea32b8-16bd-4ada-a51f-ba0e638a1457\" (UID: \"46ea32b8-16bd-4ada-a51f-ba0e638a1457\") " Oct 31 13:54:20.461948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.461799 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46ea32b8-16bd-4ada-a51f-ba0e638a1457-kube-api-access-d58b8\" (UniqueName: \"kubernetes.io/projected/46ea32b8-16bd-4ada-a51f-ba0e638a1457-kube-api-access-d58b8\") pod \"46ea32b8-16bd-4ada-a51f-ba0e638a1457\" (UID: \"46ea32b8-16bd-4ada-a51f-ba0e638a1457\") " Oct 31 13:54:20.466042 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.466010 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46ea32b8-16bd-4ada-a51f-ba0e638a1457-plugins-config-volume" pod "46ea32b8-16bd-4ada-a51f-ba0e638a1457" (UID: "46ea32b8-16bd-4ada-a51f-ba0e638a1457"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:54:20.467206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.467177 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46ea32b8-16bd-4ada-a51f-ba0e638a1457-kube-api-access-d58b8" pod "46ea32b8-16bd-4ada-a51f-ba0e638a1457" (UID: "46ea32b8-16bd-4ada-a51f-ba0e638a1457"). InnerVolumeSpecName "kube-api-access-d58b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:54:20.575686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.574059 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46ea32b8-16bd-4ada-a51f-ba0e638a1457-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:20.575686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.574100 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d58b8\" (UniqueName: \"kubernetes.io/projected/46ea32b8-16bd-4ada-a51f-ba0e638a1457-kube-api-access-d58b8\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:20.675684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.675651 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86524ce1-913b-4bc9-97b8-6adb65d9f702-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86524ce1-913b-4bc9-97b8-6adb65d9f702-plugins-config-volume\") pod \"86524ce1-913b-4bc9-97b8-6adb65d9f702\" (UID: \"86524ce1-913b-4bc9-97b8-6adb65d9f702\") " Oct 31 13:54:20.675882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.675748 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86524ce1-913b-4bc9-97b8-6adb65d9f702-kube-api-access-6rjvw\" (UniqueName: \"kubernetes.io/projected/86524ce1-913b-4bc9-97b8-6adb65d9f702-kube-api-access-6rjvw\") pod \"86524ce1-913b-4bc9-97b8-6adb65d9f702\" (UID: \"86524ce1-913b-4bc9-97b8-6adb65d9f702\") " Oct 31 13:54:20.676121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.676094 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86524ce1-913b-4bc9-97b8-6adb65d9f702-plugins-config-volume" pod "86524ce1-913b-4bc9-97b8-6adb65d9f702" (UID: "86524ce1-913b-4bc9-97b8-6adb65d9f702"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:54:20.676539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.676511 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86524ce1-913b-4bc9-97b8-6adb65d9f702-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:20.678453 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.678428 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86524ce1-913b-4bc9-97b8-6adb65d9f702-kube-api-access-6rjvw" pod "86524ce1-913b-4bc9-97b8-6adb65d9f702" (UID: "86524ce1-913b-4bc9-97b8-6adb65d9f702"). InnerVolumeSpecName "kube-api-access-6rjvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:54:20.718033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.718006 5532 generic.go:356] "Generic (PLEG): container finished" podID="86524ce1-913b-4bc9-97b8-6adb65d9f702" containerID="e01c9a4e238e20db457860e965e2e991d0ed94bb04031ffaed3bb369cc959964" exitCode=0 Oct 31 13:54:20.718199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.718052 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" Oct 31 13:54:20.718199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.718084 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" event={"ID":"86524ce1-913b-4bc9-97b8-6adb65d9f702","Type":"ContainerDied","Data":"e01c9a4e238e20db457860e965e2e991d0ed94bb04031ffaed3bb369cc959964"} Oct 31 13:54:20.718199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.718141 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt" event={"ID":"86524ce1-913b-4bc9-97b8-6adb65d9f702","Type":"ContainerDied","Data":"75dfc7e08c823a43182d359fd1111c9affe57d21925d220b349e00f7e03645ad"} Oct 31 13:54:20.718199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.718166 5532 scope.go:122] "RemoveContainer" containerID="e01c9a4e238e20db457860e965e2e991d0ed94bb04031ffaed3bb369cc959964" Oct 31 13:54:20.719762 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.719733 5532 generic.go:356] "Generic (PLEG): container finished" podID="46ea32b8-16bd-4ada-a51f-ba0e638a1457" containerID="75d076416d262d6e46f412e2c53d5005ebe6e37f77c2f9b5d118719dee5cb5b7" exitCode=0 Oct 31 13:54:20.719897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.719769 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" Oct 31 13:54:20.719897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.719805 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" event={"ID":"46ea32b8-16bd-4ada-a51f-ba0e638a1457","Type":"ContainerDied","Data":"75d076416d262d6e46f412e2c53d5005ebe6e37f77c2f9b5d118719dee5cb5b7"} Oct 31 13:54:20.719897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.719849 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4" event={"ID":"46ea32b8-16bd-4ada-a51f-ba0e638a1457","Type":"ContainerDied","Data":"a1496f7b8951c8cfe0baeb873365ace80b2fb4133c3d9e049b5107cfce08ac49"} Oct 31 13:54:20.730259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.730241 5532 scope.go:122] "RemoveContainer" containerID="e01c9a4e238e20db457860e965e2e991d0ed94bb04031ffaed3bb369cc959964" Oct 31 13:54:20.730546 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:54:20.730525 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01c9a4e238e20db457860e965e2e991d0ed94bb04031ffaed3bb369cc959964\": container with ID starting with e01c9a4e238e20db457860e965e2e991d0ed94bb04031ffaed3bb369cc959964 not found: ID does not exist" containerID="e01c9a4e238e20db457860e965e2e991d0ed94bb04031ffaed3bb369cc959964" Oct 31 13:54:20.730669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.730552 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01c9a4e238e20db457860e965e2e991d0ed94bb04031ffaed3bb369cc959964"} err="failed to get container status \"e01c9a4e238e20db457860e965e2e991d0ed94bb04031ffaed3bb369cc959964\": rpc error: code = NotFound desc = could not find container \"e01c9a4e238e20db457860e965e2e991d0ed94bb04031ffaed3bb369cc959964\": container with ID starting with e01c9a4e238e20db457860e965e2e991d0ed94bb04031ffaed3bb369cc959964 not found: ID does not exist" Oct 31 13:54:20.730669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.730567 5532 scope.go:122] "RemoveContainer" containerID="75d076416d262d6e46f412e2c53d5005ebe6e37f77c2f9b5d118719dee5cb5b7" Oct 31 13:54:20.740119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.740096 5532 scope.go:122] "RemoveContainer" containerID="75d076416d262d6e46f412e2c53d5005ebe6e37f77c2f9b5d118719dee5cb5b7" Oct 31 13:54:20.740385 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:54:20.740366 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d076416d262d6e46f412e2c53d5005ebe6e37f77c2f9b5d118719dee5cb5b7\": container with ID starting with 75d076416d262d6e46f412e2c53d5005ebe6e37f77c2f9b5d118719dee5cb5b7 not found: ID does not exist" containerID="75d076416d262d6e46f412e2c53d5005ebe6e37f77c2f9b5d118719dee5cb5b7" Oct 31 13:54:20.740451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.740390 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d076416d262d6e46f412e2c53d5005ebe6e37f77c2f9b5d118719dee5cb5b7"} err="failed to get container status \"75d076416d262d6e46f412e2c53d5005ebe6e37f77c2f9b5d118719dee5cb5b7\": rpc error: code = NotFound desc = could not find container \"75d076416d262d6e46f412e2c53d5005ebe6e37f77c2f9b5d118719dee5cb5b7\": container with ID starting with 75d076416d262d6e46f412e2c53d5005ebe6e37f77c2f9b5d118719dee5cb5b7 not found: ID does not exist" Oct 31 13:54:20.750861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.750840 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt"] Oct 31 13:54:20.764362 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.764336 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-74b5d6fd59-dt6lt"] Oct 31 13:54:20.778865 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.778828 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rjvw\" (UniqueName: \"kubernetes.io/projected/86524ce1-913b-4bc9-97b8-6adb65d9f702-kube-api-access-6rjvw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:20.782089 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.782069 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4"] Oct 31 13:54:20.785695 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:20.785677 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be675-7d6588dfb7-c4dt4"] Oct 31 13:54:21.894184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.890922 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp"] Oct 31 13:54:21.894184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.891983 5532 cpu_manager.go:414] "containerMap: removing container" podUID="46ea32b8-16bd-4ada-a51f-ba0e638a1457" containerName="payload-processing" Oct 31 13:54:21.894184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.892000 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46ea32b8-16bd-4ada-a51f-ba0e638a1457" containerName="payload-processing" Oct 31 13:54:21.894184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.892025 5532 cpu_manager.go:414] "containerMap: removing container" podUID="86524ce1-913b-4bc9-97b8-6adb65d9f702" containerName="payload-pre-processing" Oct 31 13:54:21.894184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.892035 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86524ce1-913b-4bc9-97b8-6adb65d9f702" containerName="payload-pre-processing" Oct 31 13:54:21.894184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.892197 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="46ea32b8-16bd-4ada-a51f-ba0e638a1457" containerName="payload-processing" Oct 31 13:54:21.894184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.892208 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="86524ce1-913b-4bc9-97b8-6adb65d9f702" containerName="payload-pre-processing" Oct 31 13:54:21.898227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.898206 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:21.905831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.905475 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-8a47d6-serving-cert\"" Oct 31 13:54:21.908771 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.908547 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp"] Oct 31 13:54:21.996128 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.996091 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:21.996312 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.996254 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-227pw\" (UniqueName: \"kubernetes.io/projected/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-kube-api-access-227pw\") pod \"maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:21.996417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.996398 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:21.996556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:21.996535 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-metrics-certs\") pod \"maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:22.006279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:22.006237 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46ea32b8-16bd-4ada-a51f-ba0e638a1457" path="/var/lib/kubelet/pods/46ea32b8-16bd-4ada-a51f-ba0e638a1457/volumes" Oct 31 13:54:22.006714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:22.006699 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86524ce1-913b-4bc9-97b8-6adb65d9f702" path="/var/lib/kubelet/pods/86524ce1-913b-4bc9-97b8-6adb65d9f702/volumes" Oct 31 13:54:22.098020 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:22.097982 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:22.098192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:22.098099 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-227pw\" (UniqueName: \"kubernetes.io/projected/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-kube-api-access-227pw\") pod \"maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:22.098426 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:22.098405 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:22.098630 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:22.098579 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-metrics-certs\") pod \"maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:22.099008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:22.098981 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:22.101233 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:22.101202 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-maas-api-tls\") pod \"maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:22.101233 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:22.101253 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-metrics-certs\") pod \"maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:22.114814 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:22.114778 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-227pw\" (UniqueName: \"kubernetes.io/projected/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-kube-api-access-227pw\") pod \"maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:22.216044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:22.215966 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:22.553903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:22.553869 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp"] Oct 31 13:54:22.739175 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:22.739138 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" event={"ID":"5e7f54e9-9441-4310-8129-8f7bf99f3bc6","Type":"ContainerStarted","Data":"0d8bfd9f3f929d9a42c08d5f8850e88a14840a68c0ca9a64d26be605b306d5f8"} Oct 31 13:54:23.750829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:23.750791 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" event={"ID":"5e7f54e9-9441-4310-8129-8f7bf99f3bc6","Type":"ContainerStarted","Data":"5d64728a49adb48d5a5a4cd7037a2f982f826359dec231654ae1bb48edc9949d"} Oct 31 13:54:23.751265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:23.750880 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:23.773354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:23.773300 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" podStartSLOduration=2.257163705 podStartE2EDuration="2.773285366s" totalImagesPullingTime="516.121661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:23.772149587 +0000 UTC m=+3031.917893416" watchObservedRunningTime="2026-10-31 13:54:23.773285366 +0000 UTC m=+3031.919029194" Oct 31 13:54:25.222896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.222850 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" podUID="c3b4c605-cffc-4cfa-aed6-20d22c222fa2" containerName="payload-processing" containerID="cri-o://6108f22d8c27bdc6f9a9330f40ffd8aba5b4e1c18eb020485bc249900421469b" gracePeriod=25 Oct 31 13:54:25.239720 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.239670 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" podUID="9356e238-adec-4f2e-bdd0-e9e369a660df" containerName="payload-pre-processing" containerID="cri-o://40e0750560b1316c00b792ce360dd9acaab1dc2ea1d078acb8f99380f27891ab" gracePeriod=25 Oct 31 13:54:25.574772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.574743 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" Oct 31 13:54:25.609795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.609768 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" Oct 31 13:54:25.667453 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.667424 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-kube-api-access-7rbm7\" (UniqueName: \"kubernetes.io/projected/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-kube-api-access-7rbm7\") pod \"c3b4c605-cffc-4cfa-aed6-20d22c222fa2\" (UID: \"c3b4c605-cffc-4cfa-aed6-20d22c222fa2\") " Oct 31 13:54:25.667686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.667477 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9356e238-adec-4f2e-bdd0-e9e369a660df-kube-api-access-64rjr\" (UniqueName: \"kubernetes.io/projected/9356e238-adec-4f2e-bdd0-e9e369a660df-kube-api-access-64rjr\") pod \"9356e238-adec-4f2e-bdd0-e9e369a660df\" (UID: \"9356e238-adec-4f2e-bdd0-e9e369a660df\") " Oct 31 13:54:25.667686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.667541 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9356e238-adec-4f2e-bdd0-e9e369a660df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9356e238-adec-4f2e-bdd0-e9e369a660df-plugins-config-volume\") pod \"9356e238-adec-4f2e-bdd0-e9e369a660df\" (UID: \"9356e238-adec-4f2e-bdd0-e9e369a660df\") " Oct 31 13:54:25.667686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.667645 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-plugins-config-volume\") pod \"c3b4c605-cffc-4cfa-aed6-20d22c222fa2\" (UID: \"c3b4c605-cffc-4cfa-aed6-20d22c222fa2\") " Oct 31 13:54:25.668059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.668021 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9356e238-adec-4f2e-bdd0-e9e369a660df-plugins-config-volume" pod "9356e238-adec-4f2e-bdd0-e9e369a660df" (UID: "9356e238-adec-4f2e-bdd0-e9e369a660df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:54:25.668141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.668087 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-plugins-config-volume" pod "c3b4c605-cffc-4cfa-aed6-20d22c222fa2" (UID: "c3b4c605-cffc-4cfa-aed6-20d22c222fa2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:54:25.668556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.668529 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9356e238-adec-4f2e-bdd0-e9e369a660df-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:25.668687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.668559 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:25.669723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.669689 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-kube-api-access-7rbm7" pod "c3b4c605-cffc-4cfa-aed6-20d22c222fa2" (UID: "c3b4c605-cffc-4cfa-aed6-20d22c222fa2"). InnerVolumeSpecName "kube-api-access-7rbm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:54:25.670174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.670148 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9356e238-adec-4f2e-bdd0-e9e369a660df-kube-api-access-64rjr" pod "9356e238-adec-4f2e-bdd0-e9e369a660df" (UID: "9356e238-adec-4f2e-bdd0-e9e369a660df"). InnerVolumeSpecName "kube-api-access-64rjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:54:25.778705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.770324 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rbm7\" (UniqueName: \"kubernetes.io/projected/c3b4c605-cffc-4cfa-aed6-20d22c222fa2-kube-api-access-7rbm7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:25.778705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.770360 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64rjr\" (UniqueName: \"kubernetes.io/projected/9356e238-adec-4f2e-bdd0-e9e369a660df-kube-api-access-64rjr\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:25.778705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.772063 5532 generic.go:356] "Generic (PLEG): container finished" podID="c3b4c605-cffc-4cfa-aed6-20d22c222fa2" containerID="6108f22d8c27bdc6f9a9330f40ffd8aba5b4e1c18eb020485bc249900421469b" exitCode=0 Oct 31 13:54:25.778705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.772332 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" event={"ID":"c3b4c605-cffc-4cfa-aed6-20d22c222fa2","Type":"ContainerDied","Data":"6108f22d8c27bdc6f9a9330f40ffd8aba5b4e1c18eb020485bc249900421469b"} Oct 31 13:54:25.778705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.772385 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" event={"ID":"c3b4c605-cffc-4cfa-aed6-20d22c222fa2","Type":"ContainerDied","Data":"ed9847fe1a0781bf31e8e5c388708b4572c237f49648fabfe8ed4050fa7c4ae2"} Oct 31 13:54:25.778705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.772413 5532 scope.go:122] "RemoveContainer" containerID="6108f22d8c27bdc6f9a9330f40ffd8aba5b4e1c18eb020485bc249900421469b" Oct 31 13:54:25.778705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.772539 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f" Oct 31 13:54:25.780541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.780510 5532 generic.go:356] "Generic (PLEG): container finished" podID="9356e238-adec-4f2e-bdd0-e9e369a660df" containerID="40e0750560b1316c00b792ce360dd9acaab1dc2ea1d078acb8f99380f27891ab" exitCode=0 Oct 31 13:54:25.780709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.780549 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" Oct 31 13:54:25.780709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.780625 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" event={"ID":"9356e238-adec-4f2e-bdd0-e9e369a660df","Type":"ContainerDied","Data":"40e0750560b1316c00b792ce360dd9acaab1dc2ea1d078acb8f99380f27891ab"} Oct 31 13:54:25.780709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.780676 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx" event={"ID":"9356e238-adec-4f2e-bdd0-e9e369a660df","Type":"ContainerDied","Data":"13493b1a0ea27d13e7eb8dea832b1d1cf2be77624ef942481fe2a3f46e5e3f65"} Oct 31 13:54:25.810350 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.810315 5532 scope.go:122] "RemoveContainer" containerID="6108f22d8c27bdc6f9a9330f40ffd8aba5b4e1c18eb020485bc249900421469b" Oct 31 13:54:25.810997 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:54:25.810942 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6108f22d8c27bdc6f9a9330f40ffd8aba5b4e1c18eb020485bc249900421469b\": container with ID starting with 6108f22d8c27bdc6f9a9330f40ffd8aba5b4e1c18eb020485bc249900421469b not found: ID does not exist" containerID="6108f22d8c27bdc6f9a9330f40ffd8aba5b4e1c18eb020485bc249900421469b" Oct 31 13:54:25.811200 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.811052 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6108f22d8c27bdc6f9a9330f40ffd8aba5b4e1c18eb020485bc249900421469b"} err="failed to get container status \"6108f22d8c27bdc6f9a9330f40ffd8aba5b4e1c18eb020485bc249900421469b\": rpc error: code = NotFound desc = could not find container \"6108f22d8c27bdc6f9a9330f40ffd8aba5b4e1c18eb020485bc249900421469b\": container with ID starting with 6108f22d8c27bdc6f9a9330f40ffd8aba5b4e1c18eb020485bc249900421469b not found: ID does not exist" Oct 31 13:54:25.811200 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.811082 5532 scope.go:122] "RemoveContainer" containerID="40e0750560b1316c00b792ce360dd9acaab1dc2ea1d078acb8f99380f27891ab" Oct 31 13:54:25.816088 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.816065 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f"] Oct 31 13:54:25.822447 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.822425 5532 scope.go:122] "RemoveContainer" containerID="40e0750560b1316c00b792ce360dd9acaab1dc2ea1d078acb8f99380f27891ab" Oct 31 13:54:25.822778 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:54:25.822737 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40e0750560b1316c00b792ce360dd9acaab1dc2ea1d078acb8f99380f27891ab\": container with ID starting with 40e0750560b1316c00b792ce360dd9acaab1dc2ea1d078acb8f99380f27891ab not found: ID does not exist" containerID="40e0750560b1316c00b792ce360dd9acaab1dc2ea1d078acb8f99380f27891ab" Oct 31 13:54:25.822879 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.822782 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40e0750560b1316c00b792ce360dd9acaab1dc2ea1d078acb8f99380f27891ab"} err="failed to get container status \"40e0750560b1316c00b792ce360dd9acaab1dc2ea1d078acb8f99380f27891ab\": rpc error: code = NotFound desc = could not find container \"40e0750560b1316c00b792ce360dd9acaab1dc2ea1d078acb8f99380f27891ab\": container with ID starting with 40e0750560b1316c00b792ce360dd9acaab1dc2ea1d078acb8f99380f27891ab not found: ID does not exist" Oct 31 13:54:25.823894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.823873 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a81979-54f6f9b7cf-gnk5f"] Oct 31 13:54:25.836261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.836241 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx"] Oct 31 13:54:25.848343 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:25.848317 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-58d448b487-wvgnx"] Oct 31 13:54:26.004959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:26.004926 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9356e238-adec-4f2e-bdd0-e9e369a660df" path="/var/lib/kubelet/pods/9356e238-adec-4f2e-bdd0-e9e369a660df/volumes" Oct 31 13:54:26.005431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:26.005415 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3b4c605-cffc-4cfa-aed6-20d22c222fa2" path="/var/lib/kubelet/pods/c3b4c605-cffc-4cfa-aed6-20d22c222fa2/volumes" Oct 31 13:54:29.764538 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:29.764508 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:54:31.763514 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:54:31.763472 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:54:35.936735 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:35.936697 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2"] Oct 31 13:54:35.937928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:35.937905 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c3b4c605-cffc-4cfa-aed6-20d22c222fa2" containerName="payload-processing" Oct 31 13:54:35.938002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:35.937932 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3b4c605-cffc-4cfa-aed6-20d22c222fa2" containerName="payload-processing" Oct 31 13:54:35.938002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:35.937989 5532 cpu_manager.go:414] "containerMap: removing container" podUID="9356e238-adec-4f2e-bdd0-e9e369a660df" containerName="payload-pre-processing" Oct 31 13:54:35.938002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:35.937999 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9356e238-adec-4f2e-bdd0-e9e369a660df" containerName="payload-pre-processing" Oct 31 13:54:35.938221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:35.938206 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="9356e238-adec-4f2e-bdd0-e9e369a660df" containerName="payload-pre-processing" Oct 31 13:54:35.938258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:35.938239 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3b4c605-cffc-4cfa-aed6-20d22c222fa2" containerName="payload-processing" Oct 31 13:54:35.944545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:35.944524 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" Oct 31 13:54:35.974712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:35.969396 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4be675\"" Oct 31 13:54:35.974712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:35.969763 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2"] Oct 31 13:54:35.985992 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:35.985965 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2"] Oct 31 13:54:36.010932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.010897 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2"] Oct 31 13:54:36.013802 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.013772 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" Oct 31 13:54:36.021195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.021134 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4be675-dockercfg-wljxt\"" Oct 31 13:54:36.028568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.028536 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2adbd9b5-1070-435e-81b4-f8846f493949-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2\" (UID: \"2adbd9b5-1070-435e-81b4-f8846f493949\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" Oct 31 13:54:36.028879 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.028860 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2wm\" (UniqueName: \"kubernetes.io/projected/2adbd9b5-1070-435e-81b4-f8846f493949-kube-api-access-nd2wm\") pod \"payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2\" (UID: \"2adbd9b5-1070-435e-81b4-f8846f493949\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" Oct 31 13:54:36.130303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.130248 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2wm\" (UniqueName: \"kubernetes.io/projected/2adbd9b5-1070-435e-81b4-f8846f493949-kube-api-access-nd2wm\") pod \"payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2\" (UID: \"2adbd9b5-1070-435e-81b4-f8846f493949\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" Oct 31 13:54:36.130519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.130388 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2adbd9b5-1070-435e-81b4-f8846f493949-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2\" (UID: \"2adbd9b5-1070-435e-81b4-f8846f493949\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" Oct 31 13:54:36.130519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.130455 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bjg9\" (UniqueName: \"kubernetes.io/projected/1782c792-12e9-4104-bbc4-18108c4a6e7d-kube-api-access-6bjg9\") pod \"payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2\" (UID: \"1782c792-12e9-4104-bbc4-18108c4a6e7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" Oct 31 13:54:36.130519 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.130513 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1782c792-12e9-4104-bbc4-18108c4a6e7d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2\" (UID: \"1782c792-12e9-4104-bbc4-18108c4a6e7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" Oct 31 13:54:36.143631 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.131878 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2adbd9b5-1070-435e-81b4-f8846f493949-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2\" (UID: \"2adbd9b5-1070-435e-81b4-f8846f493949\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" Oct 31 13:54:36.170165 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.169397 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2wm\" (UniqueName: \"kubernetes.io/projected/2adbd9b5-1070-435e-81b4-f8846f493949-kube-api-access-nd2wm\") pod \"payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2\" (UID: \"2adbd9b5-1070-435e-81b4-f8846f493949\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" Oct 31 13:54:36.232034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.231939 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bjg9\" (UniqueName: \"kubernetes.io/projected/1782c792-12e9-4104-bbc4-18108c4a6e7d-kube-api-access-6bjg9\") pod \"payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2\" (UID: \"1782c792-12e9-4104-bbc4-18108c4a6e7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" Oct 31 13:54:36.232220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.232052 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1782c792-12e9-4104-bbc4-18108c4a6e7d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2\" (UID: \"1782c792-12e9-4104-bbc4-18108c4a6e7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" Oct 31 13:54:36.232844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.232816 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1782c792-12e9-4104-bbc4-18108c4a6e7d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2\" (UID: \"1782c792-12e9-4104-bbc4-18108c4a6e7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" Oct 31 13:54:36.245873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.245843 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bjg9\" (UniqueName: \"kubernetes.io/projected/1782c792-12e9-4104-bbc4-18108c4a6e7d-kube-api-access-6bjg9\") pod \"payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2\" (UID: \"1782c792-12e9-4104-bbc4-18108c4a6e7d\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" Oct 31 13:54:36.319423 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.319379 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" Oct 31 13:54:36.337192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.337158 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" Oct 31 13:54:36.623235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.623142 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2"] Oct 31 13:54:36.629708 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:54:36.629677 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2adbd9b5_1070_435e_81b4_f8846f493949.slice/crio-5411f3a36b1e49d26c0df6d3f87bee3f231f3be9dbc6c0dc0d79cb4778115968 WatchSource:0}: Error finding container 5411f3a36b1e49d26c0df6d3f87bee3f231f3be9dbc6c0dc0d79cb4778115968: Status 404 returned error can't find the container with id 5411f3a36b1e49d26c0df6d3f87bee3f231f3be9dbc6c0dc0d79cb4778115968 Oct 31 13:54:36.640897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.640869 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2"] Oct 31 13:54:36.641030 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:54:36.640998 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1782c792_12e9_4104_bbc4_18108c4a6e7d.slice/crio-f91559367b5b2d8c194f5257e6935a68a815a333734e4db241d285b99de1ab6c WatchSource:0}: Error finding container f91559367b5b2d8c194f5257e6935a68a815a333734e4db241d285b99de1ab6c: Status 404 returned error can't find the container with id f91559367b5b2d8c194f5257e6935a68a815a333734e4db241d285b99de1ab6c Oct 31 13:54:36.878834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.878732 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" event={"ID":"1782c792-12e9-4104-bbc4-18108c4a6e7d","Type":"ContainerStarted","Data":"b7c8dd3eb889869571e512ed5979940049d9a2745f4886479b41d85778edcb22"} Oct 31 13:54:36.878834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.878803 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" event={"ID":"1782c792-12e9-4104-bbc4-18108c4a6e7d","Type":"ContainerStarted","Data":"f91559367b5b2d8c194f5257e6935a68a815a333734e4db241d285b99de1ab6c"} Oct 31 13:54:36.878834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.878835 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" Oct 31 13:54:36.880686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.880656 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" event={"ID":"2adbd9b5-1070-435e-81b4-f8846f493949","Type":"ContainerStarted","Data":"cc64e9cc2ec2c8440f09fe195da298558f457fae7082d59252e2bd6f5374bd2b"} Oct 31 13:54:36.880832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.880709 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" event={"ID":"2adbd9b5-1070-435e-81b4-f8846f493949","Type":"ContainerStarted","Data":"5411f3a36b1e49d26c0df6d3f87bee3f231f3be9dbc6c0dc0d79cb4778115968"} Oct 31 13:54:36.880832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.880743 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" Oct 31 13:54:36.902002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.901946 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" podStartSLOduration=1.901934147 podStartE2EDuration="1.901934147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:36.900010885 +0000 UTC m=+3045.045754713" watchObservedRunningTime="2026-10-31 13:54:36.901934147 +0000 UTC m=+3045.047677975" Oct 31 13:54:36.923714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:36.923665 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" podStartSLOduration=1.923652423 podStartE2EDuration="1.923652423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:36.921108407 +0000 UTC m=+3045.066852235" watchObservedRunningTime="2026-10-31 13:54:36.923652423 +0000 UTC m=+3045.069396251" Oct 31 13:54:38.034502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:38.034472 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986"] Oct 31 13:54:38.038234 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:38.038208 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7a526a-4g986"] Oct 31 13:54:40.005410 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.005376 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7acd287-0ae9-472a-a662-88da13fd673a" path="/var/lib/kubelet/pods/e7acd287-0ae9-472a-a662-88da13fd673a/volumes" Oct 31 13:54:40.420153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.420113 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf"] Oct 31 13:54:40.426056 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.426035 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" Oct 31 13:54:40.434862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.434829 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a81979\"" Oct 31 13:54:40.434862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.434852 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a81979-dockercfg-vj4vb\"" Oct 31 13:54:40.440002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.439973 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf"] Oct 31 13:54:40.488350 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.488296 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9"] Oct 31 13:54:40.493244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.493218 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" Oct 31 13:54:40.505152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.505124 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9"] Oct 31 13:54:40.608620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.602756 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd4wq\" (UniqueName: \"kubernetes.io/projected/51e61ea6-8c12-4113-a371-a245526ed535-kube-api-access-jd4wq\") pod \"payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf\" (UID: \"51e61ea6-8c12-4113-a371-a245526ed535\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" Oct 31 13:54:40.608620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.602818 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6dlb\" (UniqueName: \"kubernetes.io/projected/2928fcdc-f036-4614-a942-c79fc09d9227-kube-api-access-k6dlb\") pod \"payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9\" (UID: \"2928fcdc-f036-4614-a942-c79fc09d9227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" Oct 31 13:54:40.608620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.602863 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2928fcdc-f036-4614-a942-c79fc09d9227-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9\" (UID: \"2928fcdc-f036-4614-a942-c79fc09d9227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" Oct 31 13:54:40.608620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.602957 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e61ea6-8c12-4113-a371-a245526ed535-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf\" (UID: \"51e61ea6-8c12-4113-a371-a245526ed535\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" Oct 31 13:54:40.721667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.717062 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4wq\" (UniqueName: \"kubernetes.io/projected/51e61ea6-8c12-4113-a371-a245526ed535-kube-api-access-jd4wq\") pod \"payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf\" (UID: \"51e61ea6-8c12-4113-a371-a245526ed535\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" Oct 31 13:54:40.721667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.717137 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6dlb\" (UniqueName: \"kubernetes.io/projected/2928fcdc-f036-4614-a942-c79fc09d9227-kube-api-access-k6dlb\") pod \"payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9\" (UID: \"2928fcdc-f036-4614-a942-c79fc09d9227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" Oct 31 13:54:40.721667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.717191 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2928fcdc-f036-4614-a942-c79fc09d9227-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9\" (UID: \"2928fcdc-f036-4614-a942-c79fc09d9227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" Oct 31 13:54:40.721667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.717334 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e61ea6-8c12-4113-a371-a245526ed535-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf\" (UID: \"51e61ea6-8c12-4113-a371-a245526ed535\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" Oct 31 13:54:40.721667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.718777 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e61ea6-8c12-4113-a371-a245526ed535-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf\" (UID: \"51e61ea6-8c12-4113-a371-a245526ed535\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" Oct 31 13:54:40.723009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.722669 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2928fcdc-f036-4614-a942-c79fc09d9227-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9\" (UID: \"2928fcdc-f036-4614-a942-c79fc09d9227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" Oct 31 13:54:40.734045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.733989 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd4wq\" (UniqueName: \"kubernetes.io/projected/51e61ea6-8c12-4113-a371-a245526ed535-kube-api-access-jd4wq\") pod \"payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf\" (UID: \"51e61ea6-8c12-4113-a371-a245526ed535\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" Oct 31 13:54:40.734503 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.734481 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6dlb\" (UniqueName: \"kubernetes.io/projected/2928fcdc-f036-4614-a942-c79fc09d9227-kube-api-access-k6dlb\") pod \"payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9\" (UID: \"2928fcdc-f036-4614-a942-c79fc09d9227\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" Oct 31 13:54:40.744482 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.744451 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" Oct 31 13:54:40.809757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:40.809252 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" Oct 31 13:54:41.035372 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:54:41.035342 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51e61ea6_8c12_4113_a371_a245526ed535.slice/crio-409fd8e083a5eebaead86dc864f39e860452d2656e96ac6fe79e8b5a1deedc75 WatchSource:0}: Error finding container 409fd8e083a5eebaead86dc864f39e860452d2656e96ac6fe79e8b5a1deedc75: Status 404 returned error can't find the container with id 409fd8e083a5eebaead86dc864f39e860452d2656e96ac6fe79e8b5a1deedc75 Oct 31 13:54:41.035742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:41.035394 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf"] Oct 31 13:54:41.119135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:41.119102 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9"] Oct 31 13:54:41.119759 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:54:41.119733 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2928fcdc_f036_4614_a942_c79fc09d9227.slice/crio-5a39aca53257087babe6e70bd85c50eaf4d9a3fc378a20292dc20fcc44150ac6 WatchSource:0}: Error finding container 5a39aca53257087babe6e70bd85c50eaf4d9a3fc378a20292dc20fcc44150ac6: Status 404 returned error can't find the container with id 5a39aca53257087babe6e70bd85c50eaf4d9a3fc378a20292dc20fcc44150ac6 Oct 31 13:54:41.935048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:41.935005 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" event={"ID":"2928fcdc-f036-4614-a942-c79fc09d9227","Type":"ContainerStarted","Data":"79f7c435d51988369ea7683d887969bbd24f798c23aa6ff5029954a2c253be69"} Oct 31 13:54:41.935260 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:41.935062 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" event={"ID":"2928fcdc-f036-4614-a942-c79fc09d9227","Type":"ContainerStarted","Data":"5a39aca53257087babe6e70bd85c50eaf4d9a3fc378a20292dc20fcc44150ac6"} Oct 31 13:54:41.935260 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:41.935245 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" Oct 31 13:54:41.936997 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:41.936966 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" event={"ID":"51e61ea6-8c12-4113-a371-a245526ed535","Type":"ContainerStarted","Data":"4112a9c1060111b4b955dc24cf84f8c0f8559afcac15ac1571d689e158a9b662"} Oct 31 13:54:41.937132 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:41.937016 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" event={"ID":"51e61ea6-8c12-4113-a371-a245526ed535","Type":"ContainerStarted","Data":"409fd8e083a5eebaead86dc864f39e860452d2656e96ac6fe79e8b5a1deedc75"} Oct 31 13:54:41.937132 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:41.937047 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" Oct 31 13:54:41.956273 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:41.956228 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" podStartSLOduration=1.956213819 podStartE2EDuration="1.956213819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:41.954507301 +0000 UTC m=+3050.100251132" watchObservedRunningTime="2026-10-31 13:54:41.956213819 +0000 UTC m=+3050.101957647" Oct 31 13:54:41.981828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:41.981777 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" podStartSLOduration=1.9817629810000001 podStartE2EDuration="1.981762981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:41.976009902 +0000 UTC m=+3050.121753738" watchObservedRunningTime="2026-10-31 13:54:41.981762981 +0000 UTC m=+3050.127506809" Oct 31 13:54:47.892370 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:47.892330 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" Oct 31 13:54:47.892851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:47.892413 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" Oct 31 13:54:51.922106 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:51.922066 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96"] Oct 31 13:54:51.927641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:51.927606 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" Oct 31 13:54:51.945003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:51.939339 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96"] Oct 31 13:54:51.945003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:51.943467 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:54:51.962897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:51.947770 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:54:52.030548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.030513 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a08c615e-c5z96\" (UID: \"0dce842d-b481-45e1-b014-fb11bc53a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" Oct 31 13:54:52.030945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.030914 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a08c615e-c5z96\" (UID: \"0dce842d-b481-45e1-b014-fb11bc53a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" Oct 31 13:54:52.031067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.030967 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scczf\" (UniqueName: \"kubernetes.io/projected/0dce842d-b481-45e1-b014-fb11bc53a30f-kube-api-access-scczf\") pod \"maas-api-revoke-sub-a08c615e-c5z96\" (UID: \"0dce842d-b481-45e1-b014-fb11bc53a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" Oct 31 13:54:52.135119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.132869 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a08c615e-c5z96\" (UID: \"0dce842d-b481-45e1-b014-fb11bc53a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" Oct 31 13:54:52.135119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.133242 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a08c615e-c5z96\" (UID: \"0dce842d-b481-45e1-b014-fb11bc53a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" Oct 31 13:54:52.135119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.133292 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scczf\" (UniqueName: \"kubernetes.io/projected/0dce842d-b481-45e1-b014-fb11bc53a30f-kube-api-access-scczf\") pod \"maas-api-revoke-sub-a08c615e-c5z96\" (UID: \"0dce842d-b481-45e1-b014-fb11bc53a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" Oct 31 13:54:52.135119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.133861 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a08c615e-c5z96\" (UID: \"0dce842d-b481-45e1-b014-fb11bc53a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" Oct 31 13:54:52.136026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.136003 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:54:52.238404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.238298 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb"] Oct 31 13:54:52.261651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.238446 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a08c615e-c5z96\" (UID: \"0dce842d-b481-45e1-b014-fb11bc53a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" Oct 31 13:54:52.329656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.329620 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scczf\" (UniqueName: \"kubernetes.io/projected/0dce842d-b481-45e1-b014-fb11bc53a30f-kube-api-access-scczf\") pod \"maas-api-revoke-sub-a08c615e-c5z96\" (UID: \"0dce842d-b481-45e1-b014-fb11bc53a30f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" Oct 31 13:54:52.336803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.336768 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb"] Oct 31 13:54:52.337018 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.336864 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" Oct 31 13:54:52.439611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.437999 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmc8d\" (UniqueName: \"kubernetes.io/projected/5cb1fc88-f162-4f68-bd7e-522bcd20099e-kube-api-access-zmc8d\") pod \"maas-api-revoke-sub-41c4bc5b-9bqqb\" (UID: \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" Oct 31 13:54:52.439611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.438193 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41c4bc5b-9bqqb\" (UID: \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" Oct 31 13:54:52.439611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.438458 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41c4bc5b-9bqqb\" (UID: \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" Oct 31 13:54:52.604669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.603811 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41c4bc5b-9bqqb\" (UID: \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" Oct 31 13:54:52.604669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.603979 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmc8d\" (UniqueName: \"kubernetes.io/projected/5cb1fc88-f162-4f68-bd7e-522bcd20099e-kube-api-access-zmc8d\") pod \"maas-api-revoke-sub-41c4bc5b-9bqqb\" (UID: \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" Oct 31 13:54:52.604669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.604080 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41c4bc5b-9bqqb\" (UID: \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" Oct 31 13:54:52.627619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.605015 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41c4bc5b-9bqqb\" (UID: \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" Oct 31 13:54:52.636022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.632603 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmc8d\" (UniqueName: \"kubernetes.io/projected/5cb1fc88-f162-4f68-bd7e-522bcd20099e-kube-api-access-zmc8d\") pod \"maas-api-revoke-sub-41c4bc5b-9bqqb\" (UID: \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" Oct 31 13:54:52.636022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.633014 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41c4bc5b-9bqqb\" (UID: \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" Oct 31 13:54:52.639611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.638992 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:54:52.643615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:52.640649 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" Oct 31 13:54:53.024622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:53.019453 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" Oct 31 13:54:53.211852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:53.184908 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" Oct 31 13:54:53.239635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:53.235163 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" Oct 31 13:54:53.610678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:53.601886 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:54:53.731621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:53.726360 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" podUID="2257c6d8-8a08-4d03-9bad-48dd3493aecd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:54:54.334257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:54.333307 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96"] Oct 31 13:54:54.334257 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:54:54.334183 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dce842d_b481_45e1_b014_fb11bc53a30f.slice/crio-21d41db00153408067fed14f73b0bfa4dfceb27cbd22e889a10982824a2ea604 WatchSource:0}: Error finding container 21d41db00153408067fed14f73b0bfa4dfceb27cbd22e889a10982824a2ea604: Status 404 returned error can't find the container with id 21d41db00153408067fed14f73b0bfa4dfceb27cbd22e889a10982824a2ea604 Oct 31 13:54:54.538365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:54.537975 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb"] Oct 31 13:54:54.653926 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:54.653885 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" event={"ID":"0dce842d-b481-45e1-b014-fb11bc53a30f","Type":"ContainerStarted","Data":"21d41db00153408067fed14f73b0bfa4dfceb27cbd22e889a10982824a2ea604"} Oct 31 13:54:55.821665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:55.821512 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" event={"ID":"5cb1fc88-f162-4f68-bd7e-522bcd20099e","Type":"ContainerStarted","Data":"177c84385ef725a2930880c98071436de70cc9a4af60b3fec12225948bda7d69"} Oct 31 13:54:56.865478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:56.865437 5532 generic.go:356] "Generic (PLEG): container finished" podID="5cb1fc88-f162-4f68-bd7e-522bcd20099e" containerID="3819eb07f2479eeccd755bc3d4b9f9eff2518b2f4d003b9fa3c122a706b47db1" exitCode=0 Oct 31 13:54:56.866415 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:56.866371 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" event={"ID":"5cb1fc88-f162-4f68-bd7e-522bcd20099e","Type":"ContainerDied","Data":"3819eb07f2479eeccd755bc3d4b9f9eff2518b2f4d003b9fa3c122a706b47db1"} Oct 31 13:54:56.868910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:56.868884 5532 generic.go:356] "Generic (PLEG): container finished" podID="0dce842d-b481-45e1-b014-fb11bc53a30f" containerID="b1d66ee333e6fe9a4e73907d1e1822a9c9ba468b9f2606be3e567f527a4d526b" exitCode=0 Oct 31 13:54:56.869034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:56.869017 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" event={"ID":"0dce842d-b481-45e1-b014-fb11bc53a30f","Type":"ContainerDied","Data":"b1d66ee333e6fe9a4e73907d1e1822a9c9ba468b9f2606be3e567f527a4d526b"} Oct 31 13:54:58.050895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.050179 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9"] Oct 31 13:54:58.071689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.061743 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9"] Oct 31 13:54:58.071689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.062099 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" Oct 31 13:54:58.071689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.068296 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dafbce-dockercfg-t5r4p\"" Oct 31 13:54:58.071689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.068403 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dafbce\"" Oct 31 13:54:58.283638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.271707 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnnvx\" (UniqueName: \"kubernetes.io/projected/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-kube-api-access-rnnvx\") pod \"payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9\" (UID: \"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" Oct 31 13:54:58.283638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.271908 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9\" (UID: \"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" Oct 31 13:54:58.331649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.329310 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" Oct 31 13:54:58.373620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.373550 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnnvx\" (UniqueName: \"kubernetes.io/projected/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-kube-api-access-rnnvx\") pod \"payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9\" (UID: \"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" Oct 31 13:54:58.373790 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.373776 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9\" (UID: \"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" Oct 31 13:54:58.375034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.374999 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9\" (UID: \"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" Oct 31 13:54:58.412852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.412793 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnnvx\" (UniqueName: \"kubernetes.io/projected/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-kube-api-access-rnnvx\") pod \"payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9\" (UID: \"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" Oct 31 13:54:58.460972 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.460923 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" Oct 31 13:54:58.475279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.475231 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-xks-ca\") pod \"0dce842d-b481-45e1-b014-fb11bc53a30f\" (UID: \"0dce842d-b481-45e1-b014-fb11bc53a30f\") " Oct 31 13:54:58.475618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.475576 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-openshift-ca\") pod \"0dce842d-b481-45e1-b014-fb11bc53a30f\" (UID: \"0dce842d-b481-45e1-b014-fb11bc53a30f\") " Oct 31 13:54:58.475743 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.475711 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dce842d-b481-45e1-b014-fb11bc53a30f-kube-api-access-scczf\" (UniqueName: \"kubernetes.io/projected/0dce842d-b481-45e1-b014-fb11bc53a30f-kube-api-access-scczf\") pod \"0dce842d-b481-45e1-b014-fb11bc53a30f\" (UID: \"0dce842d-b481-45e1-b014-fb11bc53a30f\") " Oct 31 13:54:58.477204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.477168 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-openshift-ca" pod "0dce842d-b481-45e1-b014-fb11bc53a30f" (UID: "0dce842d-b481-45e1-b014-fb11bc53a30f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:54:58.477876 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.477851 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:58.483656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.483629 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-xks-ca" pod "0dce842d-b481-45e1-b014-fb11bc53a30f" (UID: "0dce842d-b481-45e1-b014-fb11bc53a30f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:54:58.484621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.484572 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dce842d-b481-45e1-b014-fb11bc53a30f-kube-api-access-scczf" pod "0dce842d-b481-45e1-b014-fb11bc53a30f" (UID: "0dce842d-b481-45e1-b014-fb11bc53a30f"). InnerVolumeSpecName "kube-api-access-scczf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:54:58.578777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.578737 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-xks-ca\") pod \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\" (UID: \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\") " Oct 31 13:54:58.578990 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.578933 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb1fc88-f162-4f68-bd7e-522bcd20099e-kube-api-access-zmc8d\" (UniqueName: \"kubernetes.io/projected/5cb1fc88-f162-4f68-bd7e-522bcd20099e-kube-api-access-zmc8d\") pod \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\" (UID: \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\") " Oct 31 13:54:58.578990 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.578987 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-openshift-ca\") pod \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\" (UID: \"5cb1fc88-f162-4f68-bd7e-522bcd20099e\") " Oct 31 13:54:58.579609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.579559 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-openshift-ca" pod "5cb1fc88-f162-4f68-bd7e-522bcd20099e" (UID: "5cb1fc88-f162-4f68-bd7e-522bcd20099e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:54:58.579889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.579869 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scczf\" (UniqueName: \"kubernetes.io/projected/0dce842d-b481-45e1-b014-fb11bc53a30f-kube-api-access-scczf\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:58.579975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.579895 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dce842d-b481-45e1-b014-fb11bc53a30f-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:58.579975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.579909 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:58.581941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.581912 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-xks-ca" pod "5cb1fc88-f162-4f68-bd7e-522bcd20099e" (UID: "5cb1fc88-f162-4f68-bd7e-522bcd20099e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:54:58.582244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.582223 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb1fc88-f162-4f68-bd7e-522bcd20099e-kube-api-access-zmc8d" pod "5cb1fc88-f162-4f68-bd7e-522bcd20099e" (UID: "5cb1fc88-f162-4f68-bd7e-522bcd20099e"). InnerVolumeSpecName "kube-api-access-zmc8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:54:58.682179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.682131 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmc8d\" (UniqueName: \"kubernetes.io/projected/5cb1fc88-f162-4f68-bd7e-522bcd20099e-kube-api-access-zmc8d\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:58.682179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.682180 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cb1fc88-f162-4f68-bd7e-522bcd20099e-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:54:58.703356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.703276 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" Oct 31 13:54:58.908808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.908769 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" Oct 31 13:54:58.909010 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.908901 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb" event={"ID":"5cb1fc88-f162-4f68-bd7e-522bcd20099e","Type":"ContainerDied","Data":"177c84385ef725a2930880c98071436de70cc9a4af60b3fec12225948bda7d69"} Oct 31 13:54:58.909010 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.908941 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="177c84385ef725a2930880c98071436de70cc9a4af60b3fec12225948bda7d69" Oct 31 13:54:58.915642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.910915 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" event={"ID":"0dce842d-b481-45e1-b014-fb11bc53a30f","Type":"ContainerDied","Data":"21d41db00153408067fed14f73b0bfa4dfceb27cbd22e889a10982824a2ea604"} Oct 31 13:54:58.915642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.910965 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21d41db00153408067fed14f73b0bfa4dfceb27cbd22e889a10982824a2ea604" Oct 31 13:54:58.915642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:58.911028 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96" Oct 31 13:54:59.022830 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:59.022800 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9"] Oct 31 13:54:59.024024 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:54:59.023998 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod231aefe2_c7a4_4b5d_9b9f_fdf6aa49e5b3.slice/crio-17c0416dff1bf56fc2713705a4ceb34fc6aae0627c7af9be050ed9eaab510551 WatchSource:0}: Error finding container 17c0416dff1bf56fc2713705a4ceb34fc6aae0627c7af9be050ed9eaab510551: Status 404 returned error can't find the container with id 17c0416dff1bf56fc2713705a4ceb34fc6aae0627c7af9be050ed9eaab510551 Oct 31 13:54:59.925531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:59.925312 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" event={"ID":"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3","Type":"ContainerStarted","Data":"a8d9da76db4530ac27c7fcf8fd9d66e58c61361e34d2e0e81e8fef3f89782d2d"} Oct 31 13:54:59.926067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:59.925559 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" Oct 31 13:54:59.926067 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:59.925636 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" event={"ID":"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3","Type":"ContainerStarted","Data":"17c0416dff1bf56fc2713705a4ceb34fc6aae0627c7af9be050ed9eaab510551"} Oct 31 13:54:59.951553 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:54:59.951494 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" podStartSLOduration=2.951480988 podStartE2EDuration="2.951480988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:54:59.945732226 +0000 UTC m=+3068.091476057" watchObservedRunningTime="2026-10-31 13:54:59.951480988 +0000 UTC m=+3068.097224815" Oct 31 13:55:00.122796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.122748 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:55:00.122968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.122843 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:00.579748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.579712 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:55:00.579944 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.579783 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:00.677858 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.677824 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957"] Oct 31 13:55:00.679223 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.679083 5532 cpu_manager.go:414] "containerMap: removing container" podUID="5cb1fc88-f162-4f68-bd7e-522bcd20099e" containerName="revoke-keys" Oct 31 13:55:00.679223 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.679110 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb1fc88-f162-4f68-bd7e-522bcd20099e" containerName="revoke-keys" Oct 31 13:55:00.679223 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.679155 5532 cpu_manager.go:414] "containerMap: removing container" podUID="0dce842d-b481-45e1-b014-fb11bc53a30f" containerName="revoke-keys" Oct 31 13:55:00.679223 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.679165 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dce842d-b481-45e1-b014-fb11bc53a30f" containerName="revoke-keys" Oct 31 13:55:00.679508 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.679414 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dce842d-b481-45e1-b014-fb11bc53a30f" containerName="revoke-keys" Oct 31 13:55:00.679508 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.679449 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb1fc88-f162-4f68-bd7e-522bcd20099e" containerName="revoke-keys" Oct 31 13:55:00.737054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.737012 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957"] Oct 31 13:55:00.737248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.737141 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.744413 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.744382 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.744563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.744504 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.744640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.744570 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.744640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.744614 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.744796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.744776 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51252269-ba06-454a-9a39-04f1fd3896d8-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.744897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.744847 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.744958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.744895 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zss25\" (UniqueName: \"kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-kube-api-access-zss25\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.745021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.744962 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.745100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.745084 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51252269-ba06-454a-9a39-04f1fd3896d8-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.846316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.846226 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.846316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.846262 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.846629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.846336 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51252269-ba06-454a-9a39-04f1fd3896d8-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.846629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.846369 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.846629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.846417 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zss25\" (UniqueName: \"kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-kube-api-access-zss25\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.846629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.846474 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.846629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.846586 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51252269-ba06-454a-9a39-04f1fd3896d8-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.846952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.846689 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.846952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.846773 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.846952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.846780 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.846952 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.846819 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.847216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.847192 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.847284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.847232 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51252269-ba06-454a-9a39-04f1fd3896d8-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.847284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.847236 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.849069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.849044 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.849327 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.849308 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51252269-ba06-454a-9a39-04f1fd3896d8-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.860240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.860213 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:00.860689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:00.860662 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zss25\" (UniqueName: \"kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-kube-api-access-zss25\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:01.079953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:01.079910 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:01.416377 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957"] Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:55:01.416760 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51252269_ba06_454a_9a39_04f1fd3896d8.slice/crio-0670a723837d86c79d09ba83da2854efe51ef776a2bb26b46fd5a4bc212270d6 WatchSource:0}: Error finding container 0670a723837d86c79d09ba83da2854efe51ef776a2bb26b46fd5a4bc212270d6: Status 404 returned error can't find the container with id 0670a723837d86c79d09ba83da2854efe51ef776a2bb26b46fd5a4bc212270d6 Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:01.419926 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:01.419987 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:01.420017 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:55:01.765237 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:01.942385 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" event={"ID":"51252269-ba06-454a-9a39-04f1fd3896d8","Type":"ContainerStarted","Data":"56f71a9f143c23b060c9524ec7ce3b37e9fa73f9d03d5a0dd093980feb9025db"} Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:01.942443 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" event={"ID":"51252269-ba06-454a-9a39-04f1fd3896d8","Type":"ContainerStarted","Data":"0670a723837d86c79d09ba83da2854efe51ef776a2bb26b46fd5a4bc212270d6"} Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:01.977156 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" podStartSLOduration=1.97714039 podStartE2EDuration="1.97714039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:01.972192331 +0000 UTC m=+3070.117936159" watchObservedRunningTime="2026-10-31 13:55:01.97714039 +0000 UTC m=+3070.122884218" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.080626 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.081928 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.081971 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" podUID="51252269-ba06-454a-9a39-04f1fd3896d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.274095 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf"] Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.296469 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf"] Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.296607 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.302370 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.302409 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.414306 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp52t\" (UniqueName: \"kubernetes.io/projected/40354b49-c2f2-4d25-a9e7-8be1c75c9417-kube-api-access-tp52t\") pod \"maas-api-revoke-sub-378cf721-rkrhf\" (UID: \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.414690 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-378cf721-rkrhf\" (UID: \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.414768 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-378cf721-rkrhf\" (UID: \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.516999 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp52t\" (UniqueName: \"kubernetes.io/projected/40354b49-c2f2-4d25-a9e7-8be1c75c9417-kube-api-access-tp52t\") pod \"maas-api-revoke-sub-378cf721-rkrhf\" (UID: \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.517229 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-378cf721-rkrhf\" (UID: \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.517301 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-378cf721-rkrhf\" (UID: \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.517986 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-378cf721-rkrhf\" (UID: \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.519729 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-378cf721-rkrhf\" (UID: \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.532456 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp52t\" (UniqueName: \"kubernetes.io/projected/40354b49-c2f2-4d25-a9e7-8be1c75c9417-kube-api-access-tp52t\") pod \"maas-api-revoke-sub-378cf721-rkrhf\" (UID: \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.707053 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk"] Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.712972 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.719314 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.719964 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk"] Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.823584 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwbwn\" (UniqueName: \"kubernetes.io/projected/c7b3051f-13ab-441f-902b-cce0dc8de2a8-kube-api-access-kwbwn\") pod \"maas-api-revoke-sub-b082606e-pkfjk\" (UID: \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.823855 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b082606e-pkfjk\" (UID: \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" Oct 31 13:55:02.824579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.823983 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b082606e-pkfjk\" (UID: \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" Oct 31 13:55:02.925100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.925066 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b082606e-pkfjk\" (UID: \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" Oct 31 13:55:02.925356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.925202 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwbwn\" (UniqueName: \"kubernetes.io/projected/c7b3051f-13ab-441f-902b-cce0dc8de2a8-kube-api-access-kwbwn\") pod \"maas-api-revoke-sub-b082606e-pkfjk\" (UID: \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" Oct 31 13:55:02.925356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.925342 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b082606e-pkfjk\" (UID: \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" Oct 31 13:55:02.926036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.926007 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b082606e-pkfjk\" (UID: \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" Oct 31 13:55:02.927717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.927678 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b082606e-pkfjk\" (UID: \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" Oct 31 13:55:02.946625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.939169 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwbwn\" (UniqueName: \"kubernetes.io/projected/c7b3051f-13ab-441f-902b-cce0dc8de2a8-kube-api-access-kwbwn\") pod \"maas-api-revoke-sub-b082606e-pkfjk\" (UID: \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" Oct 31 13:55:02.992527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.992501 5532 generic.go:356] "Generic (PLEG): container finished" podID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerID="92ffbb7a5f36d38e286f158c59ff493e16dc49698bfa76fe7ad991a1a7167a65" exitCode=0 Oct 31 13:55:02.992699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.992675 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" event={"ID":"dccbf2fa-9e8d-453b-ba9d-60dae20a4884","Type":"ContainerDied","Data":"92ffbb7a5f36d38e286f158c59ff493e16dc49698bfa76fe7ad991a1a7167a65"} Oct 31 13:55:02.993146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.993122 5532 scope.go:122] "RemoveContainer" containerID="92ffbb7a5f36d38e286f158c59ff493e16dc49698bfa76fe7ad991a1a7167a65" Oct 31 13:55:02.994820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.994798 5532 generic.go:356] "Generic (PLEG): container finished" podID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerID="ba38cb398433bb8c326aaf9d913b4c09baac5b5cef2b5f9c06ff230c752c872e" exitCode=0 Oct 31 13:55:02.994933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.994871 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" event={"ID":"7ee59769-d44a-4690-b5b0-aeecb07beff2","Type":"ContainerDied","Data":"ba38cb398433bb8c326aaf9d913b4c09baac5b5cef2b5f9c06ff230c752c872e"} Oct 31 13:55:02.995267 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.995239 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:02.995366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.995309 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:02.995366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.995343 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:02.996265 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.995641 5532 scope.go:122] "RemoveContainer" containerID="ba38cb398433bb8c326aaf9d913b4c09baac5b5cef2b5f9c06ff230c752c872e" Oct 31 13:55:02.997641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.997609 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:02.997721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.997690 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:02.997764 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:02.997738 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:03.077882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.077657 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf"] Oct 31 13:55:03.081636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.081057 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 31 13:55:03.081636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.081097 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" podUID="51252269-ba06-454a-9a39-04f1fd3896d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 31 13:55:03.113873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.113840 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" Oct 31 13:55:03.195866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.195842 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:55:03.432884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.432846 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp"] Oct 31 13:55:03.486981 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.486946 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" Oct 31 13:55:03.487899 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.487871 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp"] Oct 31 13:55:03.537388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.535505 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk"] Oct 31 13:55:03.612834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.612752 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pl9m\" (UniqueName: \"kubernetes.io/projected/e0009956-628d-4778-8ab0-6c6aa192e10d-kube-api-access-6pl9m\") pod \"maas-api-revoke-sub-fe48eff7-nlmrp\" (UID: \"e0009956-628d-4778-8ab0-6c6aa192e10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" Oct 31 13:55:03.614264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.613111 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe48eff7-nlmrp\" (UID: \"e0009956-628d-4778-8ab0-6c6aa192e10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" Oct 31 13:55:03.614264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.613240 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe48eff7-nlmrp\" (UID: \"e0009956-628d-4778-8ab0-6c6aa192e10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" Oct 31 13:55:03.718012 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.714560 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe48eff7-nlmrp\" (UID: \"e0009956-628d-4778-8ab0-6c6aa192e10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" Oct 31 13:55:03.718012 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.714695 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pl9m\" (UniqueName: \"kubernetes.io/projected/e0009956-628d-4778-8ab0-6c6aa192e10d-kube-api-access-6pl9m\") pod \"maas-api-revoke-sub-fe48eff7-nlmrp\" (UID: \"e0009956-628d-4778-8ab0-6c6aa192e10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" Oct 31 13:55:03.718012 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.715042 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe48eff7-nlmrp\" (UID: \"e0009956-628d-4778-8ab0-6c6aa192e10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" Oct 31 13:55:03.718298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.718245 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe48eff7-nlmrp\" (UID: \"e0009956-628d-4778-8ab0-6c6aa192e10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" Oct 31 13:55:03.718691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.718637 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe48eff7-nlmrp\" (UID: \"e0009956-628d-4778-8ab0-6c6aa192e10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" Oct 31 13:55:03.721703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.720891 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z"] Oct 31 13:55:03.728647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.728621 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pl9m\" (UniqueName: \"kubernetes.io/projected/e0009956-628d-4778-8ab0-6c6aa192e10d-kube-api-access-6pl9m\") pod \"maas-api-revoke-sub-fe48eff7-nlmrp\" (UID: \"e0009956-628d-4778-8ab0-6c6aa192e10d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" Oct 31 13:55:03.730714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.730696 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z"] Oct 31 13:55:03.730855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.730829 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" Oct 31 13:55:03.734656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.734582 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch"] Oct 31 13:55:03.736383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.736298 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-403dff-dockercfg-whgqb\"" Oct 31 13:55:03.736383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.736355 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-403dff\"" Oct 31 13:55:03.787678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.785950 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" Oct 31 13:55:03.789543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.785847 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch"] Oct 31 13:55:03.805791 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.805765 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" Oct 31 13:55:03.816338 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.816305 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2hrd\" (UniqueName: \"kubernetes.io/projected/96216deb-419a-4c46-8132-fb54d26929eb-kube-api-access-k2hrd\") pod \"payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z\" (UID: \"96216deb-419a-4c46-8132-fb54d26929eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" Oct 31 13:55:03.816482 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.816388 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9445ef10-8988-4614-80e0-7eb52d5143e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch\" (UID: \"9445ef10-8988-4614-80e0-7eb52d5143e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" Oct 31 13:55:03.816482 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.816461 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdnqb\" (UniqueName: \"kubernetes.io/projected/9445ef10-8988-4614-80e0-7eb52d5143e1-kube-api-access-hdnqb\") pod \"payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch\" (UID: \"9445ef10-8988-4614-80e0-7eb52d5143e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" Oct 31 13:55:03.816648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.816533 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96216deb-419a-4c46-8132-fb54d26929eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z\" (UID: \"96216deb-419a-4c46-8132-fb54d26929eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" Oct 31 13:55:03.918250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.918153 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2hrd\" (UniqueName: \"kubernetes.io/projected/96216deb-419a-4c46-8132-fb54d26929eb-kube-api-access-k2hrd\") pod \"payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z\" (UID: \"96216deb-419a-4c46-8132-fb54d26929eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" Oct 31 13:55:03.918895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.918860 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9445ef10-8988-4614-80e0-7eb52d5143e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch\" (UID: \"9445ef10-8988-4614-80e0-7eb52d5143e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" Oct 31 13:55:03.919039 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.918954 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdnqb\" (UniqueName: \"kubernetes.io/projected/9445ef10-8988-4614-80e0-7eb52d5143e1-kube-api-access-hdnqb\") pod \"payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch\" (UID: \"9445ef10-8988-4614-80e0-7eb52d5143e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" Oct 31 13:55:03.919111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.919033 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96216deb-419a-4c46-8132-fb54d26929eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z\" (UID: \"96216deb-419a-4c46-8132-fb54d26929eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" Oct 31 13:55:03.920183 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.920143 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96216deb-419a-4c46-8132-fb54d26929eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z\" (UID: \"96216deb-419a-4c46-8132-fb54d26929eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" Oct 31 13:55:03.925241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:03.923293 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9445ef10-8988-4614-80e0-7eb52d5143e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch\" (UID: \"9445ef10-8988-4614-80e0-7eb52d5143e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" Oct 31 13:55:04.043669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.037496 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdnqb\" (UniqueName: \"kubernetes.io/projected/9445ef10-8988-4614-80e0-7eb52d5143e1-kube-api-access-hdnqb\") pod \"payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch\" (UID: \"9445ef10-8988-4614-80e0-7eb52d5143e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" Oct 31 13:55:04.073498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.073465 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2hrd\" (UniqueName: \"kubernetes.io/projected/96216deb-419a-4c46-8132-fb54d26929eb-kube-api-access-k2hrd\") pod \"payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z\" (UID: \"96216deb-419a-4c46-8132-fb54d26929eb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" Oct 31 13:55:04.093377 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.093329 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 31 13:55:04.097782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.097730 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" podUID="51252269-ba06-454a-9a39-04f1fd3896d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 31 13:55:04.098395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.095280 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" event={"ID":"40354b49-c2f2-4d25-a9e7-8be1c75c9417","Type":"ContainerStarted","Data":"c577b7ca0c4fc628b263254d3f09a1d2e3cebfea8543b5472cb1b94f4a2a66e3"} Oct 31 13:55:04.098621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.098583 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" event={"ID":"40354b49-c2f2-4d25-a9e7-8be1c75c9417","Type":"ContainerStarted","Data":"310aefc41791907f4899e25ad4cda127b00bdef2b3907e398b5b5cfee5bf4c2d"} Oct 31 13:55:04.100206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.099438 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" Oct 31 13:55:04.120734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.107999 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc"] Oct 31 13:55:04.120734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.112489 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" Oct 31 13:55:04.120734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.119911 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" event={"ID":"dccbf2fa-9e8d-453b-ba9d-60dae20a4884","Type":"ContainerStarted","Data":"350796dabd8abc7f9e629f8689575fbe71290caf9f8d923b362475dfd3f8f107"} Oct 31 13:55:04.120734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.119964 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" event={"ID":"c7b3051f-13ab-441f-902b-cce0dc8de2a8","Type":"ContainerStarted","Data":"0bcd1a591afd951e3e909ab0194f1bf5a84a4a321f1c844485fa3e0c937f99c2"} Oct 31 13:55:04.121374 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.121331 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.130046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.129930 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-97927b-openshift-default-dockercfg-p88tk\"" Oct 31 13:55:04.132381 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.130946 5532 generic.go:356] "Generic (PLEG): container finished" podID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerID="c79c50c5d0949535c4d1acb318fdcc07c4b14bfa3d7478b34271be8a1112662e" exitCode=0 Oct 31 13:55:04.132381 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.131098 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" event={"ID":"4723d399-8244-4fd0-81f2-ce35a33441c8","Type":"ContainerDied","Data":"c79c50c5d0949535c4d1acb318fdcc07c4b14bfa3d7478b34271be8a1112662e"} Oct 31 13:55:04.134712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.134643 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qck4j\" (UniqueName: \"kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-kube-api-access-qck4j\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.134959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.134871 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istiod-ca-cert\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.135226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.135071 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.148402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.148377 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-certs\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.148583 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.148565 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-data\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.149080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.147874 5532 scope.go:122] "RemoveContainer" containerID="c79c50c5d0949535c4d1acb318fdcc07c4b14bfa3d7478b34271be8a1112662e" Oct 31 13:55:04.149866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.149845 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-podinfo\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.150715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.150689 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-token\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.150823 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.150740 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-credential-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.150823 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.150799 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-envoy\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.154563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.154532 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" event={"ID":"7ee59769-d44a-4690-b5b0-aeecb07beff2","Type":"ContainerStarted","Data":"4c31911ad78912113e42a44975329378a63de2df866fa2a104b694b394a00b60"} Oct 31 13:55:04.201768 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.162300 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" podStartSLOduration=2.162282038 podStartE2EDuration="2.162282038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:04.128005741 +0000 UTC m=+3072.273749570" watchObservedRunningTime="2026-10-31 13:55:04.162282038 +0000 UTC m=+3072.308025848" Oct 31 13:55:04.203715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.203680 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc"] Oct 31 13:55:04.206184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.206148 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:04.206344 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.206245 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:04.206344 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.206305 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.273333 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.273565 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-certs\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.273658 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-data\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.273817 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-podinfo\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.273880 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-token\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.273939 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-credential-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.274016 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-envoy\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.274223 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qck4j\" (UniqueName: \"kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-kube-api-access-qck4j\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.274441 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-data\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.274462 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istiod-ca-cert\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.274881 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.275586 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-certs\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.275642 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istiod-ca-cert\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.304204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.279049 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-credential-socket\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.320474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.316727 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-envoy\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.322644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.322347 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-podinfo\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.325627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.325456 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qck4j\" (UniqueName: \"kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-kube-api-access-qck4j\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.330264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.330218 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-token\") pod \"e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.379609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.345003 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:55:04.379609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.359323 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 13:55:04.379609 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.359376 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 13:55:04.524826 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.519421 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp"] Oct 31 13:55:04.533278 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.532563 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:04.822780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.822701 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:55:04.826072 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.825534 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 13:55:04.826072 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:04.825609 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 13:55:05.133856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:05.130877 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 31 13:55:05.133856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:05.130929 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" podUID="51252269-ba06-454a-9a39-04f1fd3896d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 31 13:55:05.164575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:05.164487 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z"] Oct 31 13:55:05.180646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:05.180585 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch"] Oct 31 13:55:05.359259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:05.359169 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 13:55:05.361963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:05.361024 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 13:55:05.823391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:05.822956 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 13:55:05.823391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:05.823018 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 13:55:05.826726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:05.826695 5532 generic.go:356] "Generic (PLEG): container finished" podID="c7b3051f-13ab-441f-902b-cce0dc8de2a8" containerID="bc960d18b7627f3fb039dd5864d0150436dbab5cb1f8b8af0733939907189c1b" exitCode=0 Oct 31 13:55:05.826991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:05.826960 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" event={"ID":"c7b3051f-13ab-441f-902b-cce0dc8de2a8","Type":"ContainerDied","Data":"bc960d18b7627f3fb039dd5864d0150436dbab5cb1f8b8af0733939907189c1b"} Oct 31 13:55:05.833826 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:05.833800 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" event={"ID":"4723d399-8244-4fd0-81f2-ce35a33441c8","Type":"ContainerStarted","Data":"b4e5e0874fbecabb974eb2b5cc9705f41c840dd35890a4e90c2d5542468c3230"} Oct 31 13:55:05.955588 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:05.955547 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" event={"ID":"96216deb-419a-4c46-8132-fb54d26929eb","Type":"ContainerStarted","Data":"284c36f2bd58db6a0e42e4e05995f46c4ddd177629403c71d80704d477264ea3"} Oct 31 13:55:06.128647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.113801 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 31 13:55:06.128647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.113884 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" podUID="51252269-ba06-454a-9a39-04f1fd3896d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 31 13:55:06.432734 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.424743 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" event={"ID":"e0009956-628d-4778-8ab0-6c6aa192e10d","Type":"ContainerStarted","Data":"e8dd24034c7e955191c933df8163197d4d4f007c05c9b483f083c81c782c546d"} Oct 31 13:55:06.517568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.517534 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 13:55:06.517854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.517825 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 13:55:06.538619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.533778 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:06.639623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.627146 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:06.815522 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.815481 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:55:06.886064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.874352 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:06.901640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.887053 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 31 13:55:06.901640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.887110 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 31 13:55:06.901640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.889578 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:06.901640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.897135 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 13:55:06.901640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.897180 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 13:55:06.915346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.915084 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:06.950617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.949067 5532 generic.go:356] "Generic (PLEG): container finished" podID="40354b49-c2f2-4d25-a9e7-8be1c75c9417" containerID="c577b7ca0c4fc628b263254d3f09a1d2e3cebfea8543b5472cb1b94f4a2a66e3" exitCode=0 Oct 31 13:55:06.961617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.952404 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" event={"ID":"40354b49-c2f2-4d25-a9e7-8be1c75c9417","Type":"ContainerDied","Data":"c577b7ca0c4fc628b263254d3f09a1d2e3cebfea8543b5472cb1b94f4a2a66e3"} Oct 31 13:55:06.961893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:06.959982 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" podUID="2257c6d8-8a08-4d03-9bad-48dd3493aecd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:07.019624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.019473 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:07.372573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.372522 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 31 13:55:07.506778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.502772 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" podUID="51252269-ba06-454a-9a39-04f1fd3896d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 31 13:55:07.506778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.479543 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 13:55:07.506778 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.502876 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 13:55:07.630715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.606692 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:55:07.637174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.637126 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 31 13:55:07.639208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.639179 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:55:07.658997 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.640132 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:07.658997 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.658375 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:07.658997 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.658648 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:07.702622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.642682 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:55:07.702622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.640718 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:07.702622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.702513 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:55:07.702951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.702799 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" podUID="0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.223:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:07.858618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.849025 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 31 13:55:07.858618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.849097 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 31 13:55:07.971618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.955024 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:07.971618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.955651 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:07.971618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.956133 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:55:07.971618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.956216 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:55:07.971618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.965506 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" podUID="2257c6d8-8a08-4d03-9bad-48dd3493aecd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:07.971618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.969784 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": context deadline exceeded" Oct 31 13:55:07.991621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.985928 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:55:07.991621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.986049 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:55:07.991621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.987180 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:07.991621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.990550 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" podUID="2257c6d8-8a08-4d03-9bad-48dd3493aecd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:07.991621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.991378 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 13:55:07.991621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:07.991415 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 13:55:08.532207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.532073 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-sngft" podUID="0b01eeee-1f42-4e8e-af23-4e228345c60f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Oct 31 13:55:08.606629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.605822 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/health\": context deadline exceeded" Oct 31 13:55:08.616624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.605740 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": context deadline exceeded" Oct 31 13:55:08.616624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.609830 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:08.656431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.630291 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:08.656431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.633994 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 13:55:08.656431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.639676 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 13:55:08.723616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.685708 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:08.723616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.713463 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" podUID="0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.223:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:08.809678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.785731 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 13:55:08.809678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.786293 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:08.883626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.879519 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 13:55:08.906336 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.906283 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 13:55:08.907084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.883770 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 31 13:55:08.908828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.908765 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" podUID="51252269-ba06-454a-9a39-04f1fd3896d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 31 13:55:08.952015 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.951931 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:55:08.963745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.963674 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:08.974895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:08.974839 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 31 13:55:09.000518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.000457 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 31 13:55:09.057388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.057348 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:55:09.057799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.057773 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:55:09.061673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.061568 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:09.422618 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:55:09.398174 5532 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.341s" Oct 31 13:55:09.447619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.438849 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:55:09.447619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.438921 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:55:09.447619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.438945 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc"] Oct 31 13:55:09.447619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.439123 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:55:09.447619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.439163 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:55:09.447619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.439198 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:55:09.447619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.414961 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 13:55:09.447619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.439261 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 13:55:09.657841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.657803 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 31 13:55:09.686814 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.686711 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 31 13:55:09.693629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.688812 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:09.694264 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.694222 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Oct 31 13:55:09.701618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.698158 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:55:09.719884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.719832 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" podUID="0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.223:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:09.739759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.739709 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:09.743617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:09.740352 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:55:10.009247 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.008715 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" podUID="2257c6d8-8a08-4d03-9bad-48dd3493aecd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:10.011697 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.011663 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:55:10.012652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.012628 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:55:10.031652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.029470 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 13:55:10.031652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.029548 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 13:55:10.077833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.077774 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:55:10.078478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.078441 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:55:10.088950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.084801 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:10.090240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.090159 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/ready\": context deadline exceeded" Oct 31 13:55:10.091635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.090428 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded" Oct 31 13:55:10.105588 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.105556 5532 generic.go:356] "Generic (PLEG): container finished" podID="e0009956-628d-4778-8ab0-6c6aa192e10d" containerID="dc05081c75714d4a08958c35a135bd5b7657fe584b82c004c55db7a21216498d" exitCode=0 Oct 31 13:55:10.315892 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.315739 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" probeResult="failure" output="Get \"https://10.132.0.52:8000/health\": context deadline exceeded" Oct 31 13:55:10.365821 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.358188 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" start-of-body= Oct 31 13:55:10.365821 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.358254 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.247:15021/healthz/ready\": dial tcp 10.132.0.247:15021: connect: connection refused" Oct 31 13:55:10.531568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.531537 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:55:10.531981 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.531962 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:10.532138 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.532118 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" event={"ID":"e0009956-628d-4778-8ab0-6c6aa192e10d","Type":"ContainerDied","Data":"dc05081c75714d4a08958c35a135bd5b7657fe584b82c004c55db7a21216498d"} Oct 31 13:55:10.532285 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.532261 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" event={"ID":"bf1757f9-2ed2-447c-a401-ed2feecfa4ca","Type":"ContainerStarted","Data":"bc75410f8a5d5f620d9b338530338074ffb024faaf5f34c50f2621a17845ce81"} Oct 31 13:55:10.532406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.532393 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" event={"ID":"9445ef10-8988-4614-80e0-7eb52d5143e1","Type":"ContainerStarted","Data":"ea925a42699a72c01d80de5d38314ceab42b8dd11c7b0f34f6044b2804374a06"} Oct 31 13:55:10.532537 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.532526 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" Oct 31 13:55:10.532738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.532726 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" Oct 31 13:55:10.532964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.532947 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" event={"ID":"9445ef10-8988-4614-80e0-7eb52d5143e1","Type":"ContainerStarted","Data":"ccfb20eba8307ac58d170123a02e2e1999812bbfe249a218c91854c7498d6224"} Oct 31 13:55:10.538707 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.538675 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:10.658618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.649832 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 31 13:55:10.658618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.649892 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 31 13:55:10.700621 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.675783 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 13:55:10.766986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.766942 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" podUID="0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.223:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:10.788654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.769115 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:10.804752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.778777 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:10.806001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:10.761414 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" podUID="e5001cfd-6512-441f-a9d9-0b566d4f06b2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:11.164677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.005750 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:11.164677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.023714 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 31 13:55:11.164677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.029187 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" podUID="2257c6d8-8a08-4d03-9bad-48dd3493aecd" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/ready\": context deadline exceeded" Oct 31 13:55:11.164677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.029889 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": context deadline exceeded" Oct 31 13:55:11.164677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.030559 5532 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/health\": context deadline exceeded" Oct 31 13:55:11.164677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.030672 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:55:11.164677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.067043 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 13:55:11.164677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.067098 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 13:55:11.164677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.080685 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": context deadline exceeded" Oct 31 13:55:11.164677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.084668 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" probeResult="failure" output="Get \"https://10.132.0.55:8000/ready\": context deadline exceeded" Oct 31 13:55:11.164677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.086874 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 13:55:11.164677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.087262 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" probeResult="failure" output="Get \"https://10.132.0.83:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:11.164677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.100825 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" Oct 31 13:55:11.246777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.166613 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" probeResult="failure" output="Get \"https://10.132.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:11.426173 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.418978 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:55:11.557792 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.557756 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 13:55:11.558087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.558070 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 13:55:11.597802 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.597771 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 13:55:11.619664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.615447 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 13:55:11.621875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.621775 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Oct 31 13:55:11.628204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.621828 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Oct 31 13:55:11.628488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.628467 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 13:55:11.670860 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.670813 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:55:11.675616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.671429 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264" Oct 31 13:55:11.676903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.676844 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:55:11.696513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.678970 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:55:11.714026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.713986 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" event={"ID":"96216deb-419a-4c46-8132-fb54d26929eb","Type":"ContainerStarted","Data":"9f297f1685947ff8dc4b623858e66253c35cef0c0cbbe293b551a5e883397f49"} Oct 31 13:55:11.766318 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.765313 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:55:11.806294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.799517 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 13:55:11.806294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.806102 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:55:11.830636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.829055 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 13:55:11.830636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.830116 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" start-of-body= Oct 31 13:55:11.830636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:11.830162 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.240:15021/healthz/ready\": dial tcp 10.132.0.240:15021: connect: connection refused" Oct 31 13:55:12.105659 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.105406 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" podStartSLOduration=9.105380522 podStartE2EDuration="9.105380522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:11.104944181 +0000 UTC m=+3079.250688010" watchObservedRunningTime="2026-10-31 13:55:12.105380522 +0000 UTC m=+3080.251124351" Oct 31 13:55:12.126182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.126106 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" podStartSLOduration=9.126084477 podStartE2EDuration="9.126084477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:11.131500171 +0000 UTC m=+3079.277244013" watchObservedRunningTime="2026-10-31 13:55:12.126084477 +0000 UTC m=+3080.271828305" Oct 31 13:55:12.396444 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.396415 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:55:12.398213 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.398192 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" Oct 31 13:55:12.425911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.425882 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" Oct 31 13:55:12.438960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.438877 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-openshift-ca\") pod \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\" (UID: \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\") " Oct 31 13:55:12.439268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.439249 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40354b49-c2f2-4d25-a9e7-8be1c75c9417-kube-api-access-tp52t\" (UniqueName: \"kubernetes.io/projected/40354b49-c2f2-4d25-a9e7-8be1c75c9417-kube-api-access-tp52t\") pod \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\" (UID: \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\") " Oct 31 13:55:12.439488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.439471 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-xks-ca\") pod \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\" (UID: \"40354b49-c2f2-4d25-a9e7-8be1c75c9417\") " Oct 31 13:55:12.471619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.469627 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-openshift-ca" pod "40354b49-c2f2-4d25-a9e7-8be1c75c9417" (UID: "40354b49-c2f2-4d25-a9e7-8be1c75c9417"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:12.512271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.512235 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-xks-ca" pod "40354b49-c2f2-4d25-a9e7-8be1c75c9417" (UID: "40354b49-c2f2-4d25-a9e7-8be1c75c9417"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:55:12.513127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.512953 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40354b49-c2f2-4d25-a9e7-8be1c75c9417-kube-api-access-tp52t" pod "40354b49-c2f2-4d25-a9e7-8be1c75c9417" (UID: "40354b49-c2f2-4d25-a9e7-8be1c75c9417"). InnerVolumeSpecName "kube-api-access-tp52t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:12.567651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.547033 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7b3051f-13ab-441f-902b-cce0dc8de2a8-kube-api-access-kwbwn\" (UniqueName: \"kubernetes.io/projected/c7b3051f-13ab-441f-902b-cce0dc8de2a8-kube-api-access-kwbwn\") pod \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\" (UID: \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\") " Oct 31 13:55:12.567927 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.566907 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7b3051f-13ab-441f-902b-cce0dc8de2a8-kube-api-access-kwbwn" pod "c7b3051f-13ab-441f-902b-cce0dc8de2a8" (UID: "c7b3051f-13ab-441f-902b-cce0dc8de2a8"). InnerVolumeSpecName "kube-api-access-kwbwn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:12.568014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.567904 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-xks-ca\") pod \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\" (UID: \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\") " Oct 31 13:55:12.576620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.568161 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-openshift-ca\") pod \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\" (UID: \"c7b3051f-13ab-441f-902b-cce0dc8de2a8\") " Oct 31 13:55:12.579529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.579254 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:12.579529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.579290 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp52t\" (UniqueName: \"kubernetes.io/projected/40354b49-c2f2-4d25-a9e7-8be1c75c9417-kube-api-access-tp52t\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:12.579529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.579306 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40354b49-c2f2-4d25-a9e7-8be1c75c9417-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:12.579529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.579322 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwbwn\" (UniqueName: \"kubernetes.io/projected/c7b3051f-13ab-441f-902b-cce0dc8de2a8-kube-api-access-kwbwn\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:12.601694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.589233 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-openshift-ca" pod "c7b3051f-13ab-441f-902b-cce0dc8de2a8" (UID: "c7b3051f-13ab-441f-902b-cce0dc8de2a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:12.607691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.607635 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-xks-ca" pod "c7b3051f-13ab-441f-902b-cce0dc8de2a8" (UID: "c7b3051f-13ab-441f-902b-cce0dc8de2a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:55:12.654503 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.644769 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" Oct 31 13:55:12.699421 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.683112 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:12.731947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.731914 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7b3051f-13ab-441f-902b-cce0dc8de2a8-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:12.794989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.794913 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" event={"ID":"c7b3051f-13ab-441f-902b-cce0dc8de2a8","Type":"ContainerDied","Data":"0bcd1a591afd951e3e909ab0194f1bf5a84a4a321f1c844485fa3e0c937f99c2"} Oct 31 13:55:12.794989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.794971 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bcd1a591afd951e3e909ab0194f1bf5a84a4a321f1c844485fa3e0c937f99c2" Oct 31 13:55:12.795248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.795042 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk" Oct 31 13:55:12.804322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.804284 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" event={"ID":"bf1757f9-2ed2-447c-a401-ed2feecfa4ca","Type":"ContainerStarted","Data":"93e6a4a97065671b72b740c635fcabddc77106537fb6a10c347078faf84b6109"} Oct 31 13:55:12.813983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.813899 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" event={"ID":"e0009956-628d-4778-8ab0-6c6aa192e10d","Type":"ContainerDied","Data":"e8dd24034c7e955191c933df8163197d4d4f007c05c9b483f083c81c782c546d"} Oct 31 13:55:12.814213 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.814194 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8dd24034c7e955191c933df8163197d4d4f007c05c9b483f083c81c782c546d" Oct 31 13:55:12.814428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.814414 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp" Oct 31 13:55:12.822837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.822803 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" event={"ID":"40354b49-c2f2-4d25-a9e7-8be1c75c9417","Type":"ContainerDied","Data":"310aefc41791907f4899e25ad4cda127b00bdef2b3907e398b5b5cfee5bf4c2d"} Oct 31 13:55:12.823089 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.823069 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="310aefc41791907f4899e25ad4cda127b00bdef2b3907e398b5b5cfee5bf4c2d" Oct 31 13:55:12.825511 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.825491 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf" Oct 31 13:55:12.832973 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.832270 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:55:12.833148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.832979 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-xks-ca\") pod \"e0009956-628d-4778-8ab0-6c6aa192e10d\" (UID: \"e0009956-628d-4778-8ab0-6c6aa192e10d\") " Oct 31 13:55:12.833148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.833094 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-openshift-ca\") pod \"e0009956-628d-4778-8ab0-6c6aa192e10d\" (UID: \"e0009956-628d-4778-8ab0-6c6aa192e10d\") " Oct 31 13:55:12.833353 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.833298 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0009956-628d-4778-8ab0-6c6aa192e10d-kube-api-access-6pl9m\" (UniqueName: \"kubernetes.io/projected/e0009956-628d-4778-8ab0-6c6aa192e10d-kube-api-access-6pl9m\") pod \"e0009956-628d-4778-8ab0-6c6aa192e10d\" (UID: \"e0009956-628d-4778-8ab0-6c6aa192e10d\") " Oct 31 13:55:12.833475 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.833368 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:55:12.833626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.833508 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:55:12.835451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.834451 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:55:12.835451 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.834552 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 13:55:12.835649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.835511 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:55:12.839577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.835887 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-openshift-ca" pod "e0009956-628d-4778-8ab0-6c6aa192e10d" (UID: "e0009956-628d-4778-8ab0-6c6aa192e10d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:12.839577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.837043 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-xks-ca" pod "e0009956-628d-4778-8ab0-6c6aa192e10d" (UID: "e0009956-628d-4778-8ab0-6c6aa192e10d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:55:12.839577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.838625 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0009956-628d-4778-8ab0-6c6aa192e10d-kube-api-access-6pl9m" pod "e0009956-628d-4778-8ab0-6c6aa192e10d" (UID: "e0009956-628d-4778-8ab0-6c6aa192e10d"). InnerVolumeSpecName "kube-api-access-6pl9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:12.937888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.937846 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pl9m\" (UniqueName: \"kubernetes.io/projected/e0009956-628d-4778-8ab0-6c6aa192e10d-kube-api-access-6pl9m\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:12.938093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.937897 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:12.938093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:12.937915 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0009956-628d-4778-8ab0-6c6aa192e10d-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:13.005034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.004928 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" podStartSLOduration=9.004909074 podStartE2EDuration="9.004909074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:13.002180823 +0000 UTC m=+3081.147924663" watchObservedRunningTime="2026-10-31 13:55:13.004909074 +0000 UTC m=+3081.150652903" Oct 31 13:55:13.119856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.119336 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv"] Oct 31 13:55:13.123551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.120740 5532 cpu_manager.go:414] "containerMap: removing container" podUID="40354b49-c2f2-4d25-a9e7-8be1c75c9417" containerName="revoke-keys" Oct 31 13:55:13.123551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.120766 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40354b49-c2f2-4d25-a9e7-8be1c75c9417" containerName="revoke-keys" Oct 31 13:55:13.123551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.120804 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e0009956-628d-4778-8ab0-6c6aa192e10d" containerName="revoke-keys" Oct 31 13:55:13.123551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.120812 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0009956-628d-4778-8ab0-6c6aa192e10d" containerName="revoke-keys" Oct 31 13:55:13.123551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.120845 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c7b3051f-13ab-441f-902b-cce0dc8de2a8" containerName="revoke-keys" Oct 31 13:55:13.123551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.120855 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7b3051f-13ab-441f-902b-cce0dc8de2a8" containerName="revoke-keys" Oct 31 13:55:13.123551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.121111 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7b3051f-13ab-441f-902b-cce0dc8de2a8" containerName="revoke-keys" Oct 31 13:55:13.123551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.121150 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="40354b49-c2f2-4d25-a9e7-8be1c75c9417" containerName="revoke-keys" Oct 31 13:55:13.123551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.121164 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0009956-628d-4778-8ab0-6c6aa192e10d" containerName="revoke-keys" Oct 31 13:55:13.127620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.126872 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" Oct 31 13:55:13.133901 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.132687 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8a47d6\"" Oct 31 13:55:13.169764 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.134578 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8a47d6-dockercfg-lmpr9\"" Oct 31 13:55:13.171857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.171257 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hs67t\" (UniqueName: \"kubernetes.io/projected/b866dadc-bcaf-4a67-8edd-beada3347fe5-kube-api-access-hs67t\") pod \"payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv\" (UID: \"b866dadc-bcaf-4a67-8edd-beada3347fe5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" Oct 31 13:55:13.171857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.171568 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b866dadc-bcaf-4a67-8edd-beada3347fe5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv\" (UID: \"b866dadc-bcaf-4a67-8edd-beada3347fe5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" Oct 31 13:55:13.189480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.188664 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv"] Oct 31 13:55:13.211660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.210095 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj"] Oct 31 13:55:13.217485 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.217202 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" Oct 31 13:55:13.218552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.218514 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj"] Oct 31 13:55:13.287075 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.286901 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b866dadc-bcaf-4a67-8edd-beada3347fe5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv\" (UID: \"b866dadc-bcaf-4a67-8edd-beada3347fe5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" Oct 31 13:55:13.287552 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.287517 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92671137-3b3e-40d4-bb9d-48eabf91adb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj\" (UID: \"92671137-3b3e-40d4-bb9d-48eabf91adb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" Oct 31 13:55:13.287748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.287718 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hs67t\" (UniqueName: \"kubernetes.io/projected/b866dadc-bcaf-4a67-8edd-beada3347fe5-kube-api-access-hs67t\") pod \"payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv\" (UID: \"b866dadc-bcaf-4a67-8edd-beada3347fe5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" Oct 31 13:55:13.287912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.287887 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkfkf\" (UniqueName: \"kubernetes.io/projected/92671137-3b3e-40d4-bb9d-48eabf91adb3-kube-api-access-tkfkf\") pod \"payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj\" (UID: \"92671137-3b3e-40d4-bb9d-48eabf91adb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" Oct 31 13:55:13.289176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.289137 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b866dadc-bcaf-4a67-8edd-beada3347fe5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv\" (UID: \"b866dadc-bcaf-4a67-8edd-beada3347fe5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" Oct 31 13:55:13.402285 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.402240 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkfkf\" (UniqueName: \"kubernetes.io/projected/92671137-3b3e-40d4-bb9d-48eabf91adb3-kube-api-access-tkfkf\") pod \"payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj\" (UID: \"92671137-3b3e-40d4-bb9d-48eabf91adb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" Oct 31 13:55:13.417163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.417125 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92671137-3b3e-40d4-bb9d-48eabf91adb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj\" (UID: \"92671137-3b3e-40d4-bb9d-48eabf91adb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" Oct 31 13:55:13.424417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.403219 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hs67t\" (UniqueName: \"kubernetes.io/projected/b866dadc-bcaf-4a67-8edd-beada3347fe5-kube-api-access-hs67t\") pod \"payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv\" (UID: \"b866dadc-bcaf-4a67-8edd-beada3347fe5\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" Oct 31 13:55:13.424816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.418205 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92671137-3b3e-40d4-bb9d-48eabf91adb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj\" (UID: \"92671137-3b3e-40d4-bb9d-48eabf91adb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" Oct 31 13:55:13.454820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.454782 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkfkf\" (UniqueName: \"kubernetes.io/projected/92671137-3b3e-40d4-bb9d-48eabf91adb3-kube-api-access-tkfkf\") pod \"payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj\" (UID: \"92671137-3b3e-40d4-bb9d-48eabf91adb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" Oct 31 13:55:13.504043 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.504009 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" Oct 31 13:55:13.535761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.535194 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:13.538493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.536958 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 31 13:55:13.538493 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.537036 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" podUID="bf1757f9-2ed2-447c-a401-ed2feecfa4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 31 13:55:13.623204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.622972 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" Oct 31 13:55:13.908866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:13.892282 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv"] Oct 31 13:55:14.011644 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:55:14.011614 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92671137_3b3e_40d4_bb9d_48eabf91adb3.slice/crio-73dddb4d91655e01baaba0289c34016bf1903fa80e685ed35cbc86b888deac43 WatchSource:0}: Error finding container 73dddb4d91655e01baaba0289c34016bf1903fa80e685ed35cbc86b888deac43: Status 404 returned error can't find the container with id 73dddb4d91655e01baaba0289c34016bf1903fa80e685ed35cbc86b888deac43 Oct 31 13:55:14.012671 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:14.012649 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj"] Oct 31 13:55:14.536171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:14.536139 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 31 13:55:14.536341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:14.536192 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" podUID="bf1757f9-2ed2-447c-a401-ed2feecfa4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 31 13:55:14.880543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:14.880101 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" event={"ID":"92671137-3b3e-40d4-bb9d-48eabf91adb3","Type":"ContainerStarted","Data":"22f6bc4e03f2067fcdedbec64254855c0f2b450c166f2e295322ffffd35dbaf3"} Oct 31 13:55:14.880543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:14.880526 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" event={"ID":"92671137-3b3e-40d4-bb9d-48eabf91adb3","Type":"ContainerStarted","Data":"73dddb4d91655e01baaba0289c34016bf1903fa80e685ed35cbc86b888deac43"} Oct 31 13:55:14.880831 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:14.880558 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" Oct 31 13:55:14.882248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:14.882223 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" event={"ID":"b866dadc-bcaf-4a67-8edd-beada3347fe5","Type":"ContainerStarted","Data":"6d5ca2b5d2bbb512e4838aaa2a29ebdd5fccc9de85efbf57aa73f0997acff1dd"} Oct 31 13:55:14.882414 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:14.882399 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" event={"ID":"b866dadc-bcaf-4a67-8edd-beada3347fe5","Type":"ContainerStarted","Data":"085dd4452797e8163c31f42df2834e360e8cf61ca275d6ac91f4ffda37d7e778"} Oct 31 13:55:14.882814 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:14.882796 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" Oct 31 13:55:14.901182 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:14.901136 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" podStartSLOduration=1.901121179 podStartE2EDuration="1.901121179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:14.900620028 +0000 UTC m=+3083.046363856" watchObservedRunningTime="2026-10-31 13:55:14.901121179 +0000 UTC m=+3083.046865007" Oct 31 13:55:14.922348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:14.922290 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" podStartSLOduration=1.922264194 podStartE2EDuration="1.922264194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:14.920956052 +0000 UTC m=+3083.066699881" watchObservedRunningTime="2026-10-31 13:55:14.922264194 +0000 UTC m=+3083.068008023" Oct 31 13:55:15.536179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:15.536146 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 31 13:55:15.536398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:15.536203 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" podUID="bf1757f9-2ed2-447c-a401-ed2feecfa4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 31 13:55:16.535474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:16.535438 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 31 13:55:16.535943 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:16.535499 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" podUID="bf1757f9-2ed2-447c-a401-ed2feecfa4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 31 13:55:17.535691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:17.535635 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" start-of-body= Oct 31 13:55:17.536116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:17.535711 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" podUID="bf1757f9-2ed2-447c-a401-ed2feecfa4ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.6:15021/healthz/ready\": dial tcp 10.132.1.6:15021: connect: connection refused" Oct 31 13:55:18.543513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:18.543477 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:18.544518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:18.544495 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:18.545770 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:18.545744 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 13:55:19.353172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:19.353136 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn"] Oct 31 13:55:21.689932 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:21.689902 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" Oct 31 13:55:22.829980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:22.829948 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" Oct 31 13:55:24.353896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.353863 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj"] Oct 31 13:55:24.358925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.358908 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" Oct 31 13:55:24.364562 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.364533 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:55:24.364562 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.364554 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:55:24.368231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.368208 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj"] Oct 31 13:55:24.439985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.439948 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mndwf\" (UniqueName: \"kubernetes.io/projected/43578379-8de5-4e72-9aa6-42880c8af833-kube-api-access-mndwf\") pod \"maas-api-revoke-sub-2909b985-d5xfj\" (UID: \"43578379-8de5-4e72-9aa6-42880c8af833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" Oct 31 13:55:24.439985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.439986 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2909b985-d5xfj\" (UID: \"43578379-8de5-4e72-9aa6-42880c8af833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" Oct 31 13:55:24.440200 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.440131 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2909b985-d5xfj\" (UID: \"43578379-8de5-4e72-9aa6-42880c8af833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" Oct 31 13:55:24.542224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.542186 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndwf\" (UniqueName: \"kubernetes.io/projected/43578379-8de5-4e72-9aa6-42880c8af833-kube-api-access-mndwf\") pod \"maas-api-revoke-sub-2909b985-d5xfj\" (UID: \"43578379-8de5-4e72-9aa6-42880c8af833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" Oct 31 13:55:24.542224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.542228 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2909b985-d5xfj\" (UID: \"43578379-8de5-4e72-9aa6-42880c8af833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" Oct 31 13:55:24.542492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.542312 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2909b985-d5xfj\" (UID: \"43578379-8de5-4e72-9aa6-42880c8af833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" Oct 31 13:55:24.543029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.543005 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2909b985-d5xfj\" (UID: \"43578379-8de5-4e72-9aa6-42880c8af833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" Oct 31 13:55:24.544693 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.544673 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2909b985-d5xfj\" (UID: \"43578379-8de5-4e72-9aa6-42880c8af833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" Oct 31 13:55:24.556991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.556961 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndwf\" (UniqueName: \"kubernetes.io/projected/43578379-8de5-4e72-9aa6-42880c8af833-kube-api-access-mndwf\") pod \"maas-api-revoke-sub-2909b985-d5xfj\" (UID: \"43578379-8de5-4e72-9aa6-42880c8af833\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" Oct 31 13:55:24.678731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:24.678696 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" Oct 31 13:55:25.003200 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:55:25.003156 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43578379_8de5_4e72_9aa6_42880c8af833.slice/crio-4a6414644e252ea4986dad6d7241d18926d3d939d9df658847b0b44089053b91 WatchSource:0}: Error finding container 4a6414644e252ea4986dad6d7241d18926d3d939d9df658847b0b44089053b91: Status 404 returned error can't find the container with id 4a6414644e252ea4986dad6d7241d18926d3d939d9df658847b0b44089053b91 Oct 31 13:55:25.003619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:25.003577 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj"] Oct 31 13:55:25.902041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:25.902004 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" Oct 31 13:55:25.903008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:25.902984 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" Oct 31 13:55:26.019028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:26.018992 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" event={"ID":"43578379-8de5-4e72-9aa6-42880c8af833","Type":"ContainerStarted","Data":"7eaa827be8b760e6c3bbda47c22d9e291d72c2f84b616ec7b75a9c55a7a8066c"} Oct 31 13:55:26.019197 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:26.019040 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" event={"ID":"43578379-8de5-4e72-9aa6-42880c8af833","Type":"ContainerStarted","Data":"4a6414644e252ea4986dad6d7241d18926d3d939d9df658847b0b44089053b91"} Oct 31 13:55:26.040121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:26.040070 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" podStartSLOduration=2.040056415 podStartE2EDuration="2.040056415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:26.037972047 +0000 UTC m=+3094.183715875" watchObservedRunningTime="2026-10-31 13:55:26.040056415 +0000 UTC m=+3094.185800244" Oct 31 13:55:27.029336 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:27.029296 5532 generic.go:356] "Generic (PLEG): container finished" podID="43578379-8de5-4e72-9aa6-42880c8af833" containerID="7eaa827be8b760e6c3bbda47c22d9e291d72c2f84b616ec7b75a9c55a7a8066c" exitCode=0 Oct 31 13:55:27.029793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:27.029377 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" event={"ID":"43578379-8de5-4e72-9aa6-42880c8af833","Type":"ContainerDied","Data":"7eaa827be8b760e6c3bbda47c22d9e291d72c2f84b616ec7b75a9c55a7a8066c"} Oct 31 13:55:28.217308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:28.217279 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" Oct 31 13:55:28.319978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:28.319901 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43578379-8de5-4e72-9aa6-42880c8af833-kube-api-access-mndwf\" (UniqueName: \"kubernetes.io/projected/43578379-8de5-4e72-9aa6-42880c8af833-kube-api-access-mndwf\") pod \"43578379-8de5-4e72-9aa6-42880c8af833\" (UID: \"43578379-8de5-4e72-9aa6-42880c8af833\") " Oct 31 13:55:28.319978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:28.319970 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-openshift-ca\") pod \"43578379-8de5-4e72-9aa6-42880c8af833\" (UID: \"43578379-8de5-4e72-9aa6-42880c8af833\") " Oct 31 13:55:28.320186 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:28.320147 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-xks-ca\") pod \"43578379-8de5-4e72-9aa6-42880c8af833\" (UID: \"43578379-8de5-4e72-9aa6-42880c8af833\") " Oct 31 13:55:28.320413 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:28.320380 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-openshift-ca" pod "43578379-8de5-4e72-9aa6-42880c8af833" (UID: "43578379-8de5-4e72-9aa6-42880c8af833"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:28.321064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:28.321032 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:28.322438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:28.322409 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-xks-ca" pod "43578379-8de5-4e72-9aa6-42880c8af833" (UID: "43578379-8de5-4e72-9aa6-42880c8af833"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:55:28.322528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:28.322487 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43578379-8de5-4e72-9aa6-42880c8af833-kube-api-access-mndwf" pod "43578379-8de5-4e72-9aa6-42880c8af833" (UID: "43578379-8de5-4e72-9aa6-42880c8af833"). InnerVolumeSpecName "kube-api-access-mndwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:28.422959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:28.422926 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mndwf\" (UniqueName: \"kubernetes.io/projected/43578379-8de5-4e72-9aa6-42880c8af833-kube-api-access-mndwf\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:28.422959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:28.422959 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43578379-8de5-4e72-9aa6-42880c8af833-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:29.050951 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:29.050924 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" Oct 31 13:55:29.051119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:29.050966 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj" event={"ID":"43578379-8de5-4e72-9aa6-42880c8af833","Type":"ContainerDied","Data":"4a6414644e252ea4986dad6d7241d18926d3d939d9df658847b0b44089053b91"} Oct 31 13:55:29.051119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:29.051018 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a6414644e252ea4986dad6d7241d18926d3d939d9df658847b0b44089053b91" Oct 31 13:55:29.914354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:29.914319 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj"] Oct 31 13:55:29.915232 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:29.915206 5532 cpu_manager.go:414] "containerMap: removing container" podUID="43578379-8de5-4e72-9aa6-42880c8af833" containerName="revoke-keys" Oct 31 13:55:29.915232 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:29.915225 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43578379-8de5-4e72-9aa6-42880c8af833" containerName="revoke-keys" Oct 31 13:55:29.915413 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:29.915389 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="43578379-8de5-4e72-9aa6-42880c8af833" containerName="revoke-keys" Oct 31 13:55:29.919548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:29.919530 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" Oct 31 13:55:29.925425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:29.925399 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:55:29.926019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:29.925997 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:55:29.928527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:29.928505 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj"] Oct 31 13:55:30.049219 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:30.049187 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj\" (UID: \"3293093d-6ff2-4349-9d1d-f82bb31baf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" Oct 31 13:55:30.049418 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:30.049230 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8654n\" (UniqueName: \"kubernetes.io/projected/3293093d-6ff2-4349-9d1d-f82bb31baf19-kube-api-access-8654n\") pod \"maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj\" (UID: \"3293093d-6ff2-4349-9d1d-f82bb31baf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" Oct 31 13:55:30.049480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:30.049424 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj\" (UID: \"3293093d-6ff2-4349-9d1d-f82bb31baf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" Oct 31 13:55:30.150751 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:30.150717 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj\" (UID: \"3293093d-6ff2-4349-9d1d-f82bb31baf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" Oct 31 13:55:30.150920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:30.150863 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj\" (UID: \"3293093d-6ff2-4349-9d1d-f82bb31baf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" Oct 31 13:55:30.150920 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:30.150890 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8654n\" (UniqueName: \"kubernetes.io/projected/3293093d-6ff2-4349-9d1d-f82bb31baf19-kube-api-access-8654n\") pod \"maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj\" (UID: \"3293093d-6ff2-4349-9d1d-f82bb31baf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" Oct 31 13:55:30.151446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:30.151420 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj\" (UID: \"3293093d-6ff2-4349-9d1d-f82bb31baf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" Oct 31 13:55:30.153201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:30.153179 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj\" (UID: \"3293093d-6ff2-4349-9d1d-f82bb31baf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" Oct 31 13:55:30.163664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:30.163635 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8654n\" (UniqueName: \"kubernetes.io/projected/3293093d-6ff2-4349-9d1d-f82bb31baf19-kube-api-access-8654n\") pod \"maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj\" (UID: \"3293093d-6ff2-4349-9d1d-f82bb31baf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" Oct 31 13:55:30.237270 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:30.237201 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" Oct 31 13:55:30.526155 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:55:30.526115 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3293093d_6ff2_4349_9d1d_f82bb31baf19.slice/crio-a5707bc33c6db3249ddfefc37554ebfd85088149f3a80a7cfd2e7314868859dd WatchSource:0}: Error finding container a5707bc33c6db3249ddfefc37554ebfd85088149f3a80a7cfd2e7314868859dd: Status 404 returned error can't find the container with id a5707bc33c6db3249ddfefc37554ebfd85088149f3a80a7cfd2e7314868859dd Oct 31 13:55:30.526989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:30.526961 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj"] Oct 31 13:55:31.071380 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:31.071345 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" event={"ID":"3293093d-6ff2-4349-9d1d-f82bb31baf19","Type":"ContainerStarted","Data":"003db0eb79a064442fecbd645922acaedcc9c9694b67a92baf925022b380dc94"} Oct 31 13:55:31.071775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:31.071401 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" event={"ID":"3293093d-6ff2-4349-9d1d-f82bb31baf19","Type":"ContainerStarted","Data":"a5707bc33c6db3249ddfefc37554ebfd85088149f3a80a7cfd2e7314868859dd"} Oct 31 13:55:31.096310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:31.096262 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" podStartSLOduration=2.096248707 podStartE2EDuration="2.096248707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:31.090374193 +0000 UTC m=+3099.236118021" watchObservedRunningTime="2026-10-31 13:55:31.096248707 +0000 UTC m=+3099.241992550" Oct 31 13:55:31.721022 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:55:31.720986 5532 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3293093d_6ff2_4349_9d1d_f82bb31baf19.slice/crio-003db0eb79a064442fecbd645922acaedcc9c9694b67a92baf925022b380dc94.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:55:31.763643 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:55:31.763604 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:55:32.083389 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:32.083352 5532 generic.go:356] "Generic (PLEG): container finished" podID="3293093d-6ff2-4349-9d1d-f82bb31baf19" containerID="003db0eb79a064442fecbd645922acaedcc9c9694b67a92baf925022b380dc94" exitCode=0 Oct 31 13:55:32.083804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:32.083437 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" event={"ID":"3293093d-6ff2-4349-9d1d-f82bb31baf19","Type":"ContainerDied","Data":"003db0eb79a064442fecbd645922acaedcc9c9694b67a92baf925022b380dc94"} Oct 31 13:55:33.288581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:33.288553 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" Oct 31 13:55:33.314281 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:33.314243 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3293093d-6ff2-4349-9d1d-f82bb31baf19-kube-api-access-8654n\" (UniqueName: \"kubernetes.io/projected/3293093d-6ff2-4349-9d1d-f82bb31baf19-kube-api-access-8654n\") pod \"3293093d-6ff2-4349-9d1d-f82bb31baf19\" (UID: \"3293093d-6ff2-4349-9d1d-f82bb31baf19\") " Oct 31 13:55:33.314454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:33.314437 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-openshift-ca\") pod \"3293093d-6ff2-4349-9d1d-f82bb31baf19\" (UID: \"3293093d-6ff2-4349-9d1d-f82bb31baf19\") " Oct 31 13:55:33.314509 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:33.314478 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-xks-ca\") pod \"3293093d-6ff2-4349-9d1d-f82bb31baf19\" (UID: \"3293093d-6ff2-4349-9d1d-f82bb31baf19\") " Oct 31 13:55:33.314882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:33.314835 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-openshift-ca" pod "3293093d-6ff2-4349-9d1d-f82bb31baf19" (UID: "3293093d-6ff2-4349-9d1d-f82bb31baf19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:33.315297 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:33.315276 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:33.316624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:33.316571 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-xks-ca" pod "3293093d-6ff2-4349-9d1d-f82bb31baf19" (UID: "3293093d-6ff2-4349-9d1d-f82bb31baf19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:55:33.316878 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:33.316858 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3293093d-6ff2-4349-9d1d-f82bb31baf19-kube-api-access-8654n" pod "3293093d-6ff2-4349-9d1d-f82bb31baf19" (UID: "3293093d-6ff2-4349-9d1d-f82bb31baf19"). InnerVolumeSpecName "kube-api-access-8654n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:33.417289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:33.417254 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8654n\" (UniqueName: \"kubernetes.io/projected/3293093d-6ff2-4349-9d1d-f82bb31baf19-kube-api-access-8654n\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:33.417289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:33.417285 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3293093d-6ff2-4349-9d1d-f82bb31baf19-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:34.104181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.104144 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" event={"ID":"3293093d-6ff2-4349-9d1d-f82bb31baf19","Type":"ContainerDied","Data":"a5707bc33c6db3249ddfefc37554ebfd85088149f3a80a7cfd2e7314868859dd"} Oct 31 13:55:34.104394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.104204 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5707bc33c6db3249ddfefc37554ebfd85088149f3a80a7cfd2e7314868859dd" Oct 31 13:55:34.104394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.104155 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj" Oct 31 13:55:34.375367 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.375289 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" podUID="0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" containerName="main" containerID="cri-o://95b0e220e341523aa6f97af00e31e396e6fbf7ea7d271bbf3326f9e139c41ff0" gracePeriod=45 Oct 31 13:55:34.505419 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.505388 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp"] Oct 31 13:55:34.506776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.506736 5532 cpu_manager.go:414] "containerMap: removing container" podUID="3293093d-6ff2-4349-9d1d-f82bb31baf19" containerName="revoke-keys" Oct 31 13:55:34.506776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.506761 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3293093d-6ff2-4349-9d1d-f82bb31baf19" containerName="revoke-keys" Oct 31 13:55:34.507062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.507047 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="3293093d-6ff2-4349-9d1d-f82bb31baf19" containerName="revoke-keys" Oct 31 13:55:34.511873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.511849 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.512004 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.511921 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp"] Oct 31 13:55:34.516586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.516560 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a39e7e\"/\"kube-root-ca.crt\"" Oct 31 13:55:34.516732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.516720 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a39e7e\"/\"sub-test-model-a39e7e-kserve-self-signed-certs\"" Oct 31 13:55:34.516804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.516757 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-a39e7e\"/\"openshift-service-ca.crt\"" Oct 31 13:55:34.530843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.530816 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-a39e7e\"/\"default-dockercfg-scjgf\"" Oct 31 13:55:34.542893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.536280 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-model-cache\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.542893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.536347 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-home\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.542893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.536390 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tmp-dir\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.542893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.536472 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-dshm\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.542893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.536527 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxlkp\" (UniqueName: \"kubernetes.io/projected/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-kube-api-access-bxlkp\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.542893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.536711 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tls-certs\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.638620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.638522 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tls-certs\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.638816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.638683 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-model-cache\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.638816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.638721 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-home\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.638816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.638750 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tmp-dir\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.638816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.638800 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-dshm\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.639048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.638840 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxlkp\" (UniqueName: \"kubernetes.io/projected/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-kube-api-access-bxlkp\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.639150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.639127 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-model-cache\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.639211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.639184 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-home\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.639332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.639307 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tmp-dir\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.641387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.641247 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-dshm\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.641387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.641343 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tls-certs\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.652094 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.652074 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxlkp\" (UniqueName: \"kubernetes.io/projected/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-kube-api-access-bxlkp\") pod \"sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.689423 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.689394 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:55:34.748618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.747984 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-model-cache\") pod \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " Oct 31 13:55:34.748618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.748170 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-kube-api-access-5d7fk\" (UniqueName: \"kubernetes.io/projected/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-kube-api-access-5d7fk\") pod \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " Oct 31 13:55:34.748618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.748267 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-dshm\") pod \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " Oct 31 13:55:34.748618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.748306 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tls-certs\") pod \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " Oct 31 13:55:34.748618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.748341 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-home\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-home\") pod \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " Oct 31 13:55:34.748618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.748452 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tmp-dir\") pod \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\" (UID: \"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8\") " Oct 31 13:55:34.771616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.768817 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-model-cache" pod "0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" (UID: "0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:55:34.771616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.769931 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-home" pod "0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" (UID: "0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:55:34.824250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.824138 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tmp-dir" pod "0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" (UID: "0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:55:34.835268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.835196 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-kube-api-access-5d7fk" pod "0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" (UID: "0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8"). InnerVolumeSpecName "kube-api-access-5d7fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:34.837146 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.837112 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-dshm" pod "0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" (UID: "0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:55:34.837281 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.837182 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tls-certs" pod "0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" (UID: "0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:55:34.840692 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.840651 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t"] Oct 31 13:55:34.842128 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.842100 5532 cpu_manager.go:414] "containerMap: removing container" podUID="0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" containerName="main" Oct 31 13:55:34.842263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.842131 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" containerName="main" Oct 31 13:55:34.842407 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.842389 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" containerName="main" Oct 31 13:55:34.844656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.844629 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:34.851732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.851704 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:34.851841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.851735 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d7fk\" (UniqueName: \"kubernetes.io/projected/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-kube-api-access-5d7fk\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:34.851841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.851749 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:34.851841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.851764 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:34.851841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.851776 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:34.851841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.851788 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:34.853672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.853650 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t"] Oct 31 13:55:34.853781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.853764 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" Oct 31 13:55:34.862480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.862094 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:55:34.862480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.862260 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:55:34.953077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.953046 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5d97f4b-7k27t\" (UID: \"a5335d46-2571-41b5-988e-621350b8ffac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" Oct 31 13:55:34.953248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.953092 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5d97f4b-7k27t\" (UID: \"a5335d46-2571-41b5-988e-621350b8ffac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" Oct 31 13:55:34.953425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:34.953399 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnqgc\" (UniqueName: \"kubernetes.io/projected/a5335d46-2571-41b5-988e-621350b8ffac-kube-api-access-hnqgc\") pod \"maas-api-revoke-sub-b5d97f4b-7k27t\" (UID: \"a5335d46-2571-41b5-988e-621350b8ffac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" Oct 31 13:55:35.054626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.054577 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnqgc\" (UniqueName: \"kubernetes.io/projected/a5335d46-2571-41b5-988e-621350b8ffac-kube-api-access-hnqgc\") pod \"maas-api-revoke-sub-b5d97f4b-7k27t\" (UID: \"a5335d46-2571-41b5-988e-621350b8ffac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" Oct 31 13:55:35.054834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.054815 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5d97f4b-7k27t\" (UID: \"a5335d46-2571-41b5-988e-621350b8ffac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" Oct 31 13:55:35.054914 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.054864 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5d97f4b-7k27t\" (UID: \"a5335d46-2571-41b5-988e-621350b8ffac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" Oct 31 13:55:35.055537 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.055514 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b5d97f4b-7k27t\" (UID: \"a5335d46-2571-41b5-988e-621350b8ffac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" Oct 31 13:55:35.057241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.057216 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b5d97f4b-7k27t\" (UID: \"a5335d46-2571-41b5-988e-621350b8ffac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" Oct 31 13:55:35.067686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.067662 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnqgc\" (UniqueName: \"kubernetes.io/projected/a5335d46-2571-41b5-988e-621350b8ffac-kube-api-access-hnqgc\") pod \"maas-api-revoke-sub-b5d97f4b-7k27t\" (UID: \"a5335d46-2571-41b5-988e-621350b8ffac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" Oct 31 13:55:35.113954 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.113923 5532 generic.go:356] "Generic (PLEG): container finished" podID="0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" containerID="95b0e220e341523aa6f97af00e31e396e6fbf7ea7d271bbf3326f9e139c41ff0" exitCode=0 Oct 31 13:55:35.114110 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.113963 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" Oct 31 13:55:35.114110 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.114007 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" event={"ID":"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8","Type":"ContainerDied","Data":"95b0e220e341523aa6f97af00e31e396e6fbf7ea7d271bbf3326f9e139c41ff0"} Oct 31 13:55:35.114110 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.114052 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn" event={"ID":"0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8","Type":"ContainerDied","Data":"75da5561d501b6a2fec65e5178c4dfff14813c670c52117c058b1cfa3cd61d13"} Oct 31 13:55:35.114110 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.114072 5532 scope.go:122] "RemoveContainer" containerID="95b0e220e341523aa6f97af00e31e396e6fbf7ea7d271bbf3326f9e139c41ff0" Oct 31 13:55:35.126626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.126606 5532 scope.go:122] "RemoveContainer" containerID="95b0e220e341523aa6f97af00e31e396e6fbf7ea7d271bbf3326f9e139c41ff0" Oct 31 13:55:35.126915 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:55:35.126895 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95b0e220e341523aa6f97af00e31e396e6fbf7ea7d271bbf3326f9e139c41ff0\": container with ID starting with 95b0e220e341523aa6f97af00e31e396e6fbf7ea7d271bbf3326f9e139c41ff0 not found: ID does not exist" containerID="95b0e220e341523aa6f97af00e31e396e6fbf7ea7d271bbf3326f9e139c41ff0" Oct 31 13:55:35.126970 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.126924 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95b0e220e341523aa6f97af00e31e396e6fbf7ea7d271bbf3326f9e139c41ff0"} err="failed to get container status \"95b0e220e341523aa6f97af00e31e396e6fbf7ea7d271bbf3326f9e139c41ff0\": rpc error: code = NotFound desc = could not find container \"95b0e220e341523aa6f97af00e31e396e6fbf7ea7d271bbf3326f9e139c41ff0\": container with ID starting with 95b0e220e341523aa6f97af00e31e396e6fbf7ea7d271bbf3326f9e139c41ff0 not found: ID does not exist" Oct 31 13:55:35.153725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.153691 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn"] Oct 31 13:55:35.159023 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:55:35.158995 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9094dd3_ce3e_4ee4_a8e2_8af6350f6dea.slice/crio-fa33590340f7bbf71af163b7b5188ced2a1a2bace80ebdb8ebb3946b57fafb64 WatchSource:0}: Error finding container fa33590340f7bbf71af163b7b5188ced2a1a2bace80ebdb8ebb3946b57fafb64: Status 404 returned error can't find the container with id fa33590340f7bbf71af163b7b5188ced2a1a2bace80ebdb8ebb3946b57fafb64 Oct 31 13:55:35.160018 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.159997 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp"] Oct 31 13:55:35.164726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.164704 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-35d13e/auth-test-model-35d13e-kserve-84c7f6f6fc-5pgjn"] Oct 31 13:55:35.180712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.180690 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" Oct 31 13:55:35.475640 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:55:35.475612 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5335d46_2571_41b5_988e_621350b8ffac.slice/crio-8bced07b5c32a9e6ea542ba2ee648c399841278e9ff7ad94849af1a39fdb98b0 WatchSource:0}: Error finding container 8bced07b5c32a9e6ea542ba2ee648c399841278e9ff7ad94849af1a39fdb98b0: Status 404 returned error can't find the container with id 8bced07b5c32a9e6ea542ba2ee648c399841278e9ff7ad94849af1a39fdb98b0 Oct 31 13:55:35.475980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:35.475898 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t"] Oct 31 13:55:36.007504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:36.007468 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8" path="/var/lib/kubelet/pods/0c2fd3e5-38a5-4872-9ae8-9856d24cc3f8/volumes" Oct 31 13:55:36.125340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:36.125247 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" event={"ID":"a5335d46-2571-41b5-988e-621350b8ffac","Type":"ContainerStarted","Data":"89d55aefc5feb53d811a83c9397abb0ba55c5531977e5f177e9c4a1681aaa6e1"} Oct 31 13:55:36.125340 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:36.125314 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" event={"ID":"a5335d46-2571-41b5-988e-621350b8ffac","Type":"ContainerStarted","Data":"8bced07b5c32a9e6ea542ba2ee648c399841278e9ff7ad94849af1a39fdb98b0"} Oct 31 13:55:36.127054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:36.127024 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" event={"ID":"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea","Type":"ContainerStarted","Data":"933644d748b13a2a2f9bcb9808ce183d2ff87e1f242493f93a19e50c962f2a78"} Oct 31 13:55:36.127204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:36.127064 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" event={"ID":"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea","Type":"ContainerStarted","Data":"fa33590340f7bbf71af163b7b5188ced2a1a2bace80ebdb8ebb3946b57fafb64"} Oct 31 13:55:36.149249 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:36.149197 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" podStartSLOduration=2.149182568 podStartE2EDuration="2.149182568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:36.145745746 +0000 UTC m=+3104.291489574" watchObservedRunningTime="2026-10-31 13:55:36.149182568 +0000 UTC m=+3104.294926393" Oct 31 13:55:36.168478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:36.168430 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" podStartSLOduration=2.168417017 podStartE2EDuration="2.168417017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:36.166832509 +0000 UTC m=+3104.312576337" watchObservedRunningTime="2026-10-31 13:55:36.168417017 +0000 UTC m=+3104.314160844" Oct 31 13:55:36.845372 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:36.845265 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:37.138029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:37.137932 5532 generic.go:356] "Generic (PLEG): container finished" podID="a5335d46-2571-41b5-988e-621350b8ffac" containerID="89d55aefc5feb53d811a83c9397abb0ba55c5531977e5f177e9c4a1681aaa6e1" exitCode=0 Oct 31 13:55:37.138029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:37.138015 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" event={"ID":"a5335d46-2571-41b5-988e-621350b8ffac","Type":"ContainerDied","Data":"89d55aefc5feb53d811a83c9397abb0ba55c5531977e5f177e9c4a1681aaa6e1"} Oct 31 13:55:38.333897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:38.333871 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" Oct 31 13:55:38.416686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:38.416565 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-openshift-ca\") pod \"a5335d46-2571-41b5-988e-621350b8ffac\" (UID: \"a5335d46-2571-41b5-988e-621350b8ffac\") " Oct 31 13:55:38.416686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:38.416647 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-xks-ca\") pod \"a5335d46-2571-41b5-988e-621350b8ffac\" (UID: \"a5335d46-2571-41b5-988e-621350b8ffac\") " Oct 31 13:55:38.416893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:38.416803 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5335d46-2571-41b5-988e-621350b8ffac-kube-api-access-hnqgc\" (UniqueName: \"kubernetes.io/projected/a5335d46-2571-41b5-988e-621350b8ffac-kube-api-access-hnqgc\") pod \"a5335d46-2571-41b5-988e-621350b8ffac\" (UID: \"a5335d46-2571-41b5-988e-621350b8ffac\") " Oct 31 13:55:38.416939 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:38.416889 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-openshift-ca" pod "a5335d46-2571-41b5-988e-621350b8ffac" (UID: "a5335d46-2571-41b5-988e-621350b8ffac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:38.417356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:38.417337 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:38.418763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:38.418734 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-xks-ca" pod "a5335d46-2571-41b5-988e-621350b8ffac" (UID: "a5335d46-2571-41b5-988e-621350b8ffac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:55:38.418881 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:38.418860 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5335d46-2571-41b5-988e-621350b8ffac-kube-api-access-hnqgc" pod "a5335d46-2571-41b5-988e-621350b8ffac" (UID: "a5335d46-2571-41b5-988e-621350b8ffac"). InnerVolumeSpecName "kube-api-access-hnqgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:38.519246 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:38.519204 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5335d46-2571-41b5-988e-621350b8ffac-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:38.519246 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:38.519242 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnqgc\" (UniqueName: \"kubernetes.io/projected/a5335d46-2571-41b5-988e-621350b8ffac-kube-api-access-hnqgc\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:39.156427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:39.156397 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" Oct 31 13:55:39.156626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:39.156437 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t" event={"ID":"a5335d46-2571-41b5-988e-621350b8ffac","Type":"ContainerDied","Data":"8bced07b5c32a9e6ea542ba2ee648c399841278e9ff7ad94849af1a39fdb98b0"} Oct 31 13:55:39.156626 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:39.156492 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bced07b5c32a9e6ea542ba2ee648c399841278e9ff7ad94849af1a39fdb98b0" Oct 31 13:55:39.644157 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:39.644124 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw"] Oct 31 13:55:42.458249 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:42.458210 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk"] Oct 31 13:55:42.458779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:42.458505 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" podUID="9eef8cf1-7c9e-489c-8ec6-0b5175e28e01" containerName="payload-processing" containerID="cri-o://9f046bd77be5df61c394c55e4747905839ecc799100720dbe8c5fe1eafddb9e9" gracePeriod=30 Oct 31 13:55:42.476398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:42.476365 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w"] Oct 31 13:55:42.476861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:42.476826 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" podUID="7981c832-1940-492c-a03a-96dd7a8d473e" containerName="payload-pre-processing" containerID="cri-o://b9475fd85cd13f152275e658d5d399c779ab1f3e35f30dcaaaf21b6d28c718bb" gracePeriod=30 Oct 31 13:55:42.926040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:42.925649 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" Oct 31 13:55:42.933111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:42.933079 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj"] Oct 31 13:55:42.940990 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:42.940957 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-20b8a4-6ztlj"] Oct 31 13:55:42.994355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:42.994179 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-plugins-config-volume\") pod \"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01\" (UID: \"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01\") " Oct 31 13:55:42.994355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:42.994253 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-kube-api-access-69qxq\" (UniqueName: \"kubernetes.io/projected/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-kube-api-access-69qxq\") pod \"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01\" (UID: \"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01\") " Oct 31 13:55:42.994843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:42.994813 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-plugins-config-volume" pod "9eef8cf1-7c9e-489c-8ec6-0b5175e28e01" (UID: "9eef8cf1-7c9e-489c-8ec6-0b5175e28e01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:42.997025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:42.996979 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-kube-api-access-69qxq" pod "9eef8cf1-7c9e-489c-8ec6-0b5175e28e01" (UID: "9eef8cf1-7c9e-489c-8ec6-0b5175e28e01"). InnerVolumeSpecName "kube-api-access-69qxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:43.000242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.000212 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" Oct 31 13:55:43.095967 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.095816 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7981c832-1940-492c-a03a-96dd7a8d473e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7981c832-1940-492c-a03a-96dd7a8d473e-plugins-config-volume\") pod \"7981c832-1940-492c-a03a-96dd7a8d473e\" (UID: \"7981c832-1940-492c-a03a-96dd7a8d473e\") " Oct 31 13:55:43.096194 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.096164 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7981c832-1940-492c-a03a-96dd7a8d473e-kube-api-access-26txg\" (UniqueName: \"kubernetes.io/projected/7981c832-1940-492c-a03a-96dd7a8d473e-kube-api-access-26txg\") pod \"7981c832-1940-492c-a03a-96dd7a8d473e\" (UID: \"7981c832-1940-492c-a03a-96dd7a8d473e\") " Oct 31 13:55:43.096291 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.096268 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7981c832-1940-492c-a03a-96dd7a8d473e-plugins-config-volume" pod "7981c832-1940-492c-a03a-96dd7a8d473e" (UID: "7981c832-1940-492c-a03a-96dd7a8d473e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:43.096810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.096787 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:43.096810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.096808 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69qxq\" (UniqueName: \"kubernetes.io/projected/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01-kube-api-access-69qxq\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:43.097003 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.096823 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7981c832-1940-492c-a03a-96dd7a8d473e-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:43.098410 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.098388 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7981c832-1940-492c-a03a-96dd7a8d473e-kube-api-access-26txg" pod "7981c832-1940-492c-a03a-96dd7a8d473e" (UID: "7981c832-1940-492c-a03a-96dd7a8d473e"). InnerVolumeSpecName "kube-api-access-26txg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:43.198187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.198155 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26txg\" (UniqueName: \"kubernetes.io/projected/7981c832-1940-492c-a03a-96dd7a8d473e-kube-api-access-26txg\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:43.199089 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.199066 5532 generic.go:356] "Generic (PLEG): container finished" podID="7981c832-1940-492c-a03a-96dd7a8d473e" containerID="b9475fd85cd13f152275e658d5d399c779ab1f3e35f30dcaaaf21b6d28c718bb" exitCode=0 Oct 31 13:55:43.199168 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.199113 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" Oct 31 13:55:43.199168 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.199147 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" event={"ID":"7981c832-1940-492c-a03a-96dd7a8d473e","Type":"ContainerDied","Data":"b9475fd85cd13f152275e658d5d399c779ab1f3e35f30dcaaaf21b6d28c718bb"} Oct 31 13:55:43.199252 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.199205 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w" event={"ID":"7981c832-1940-492c-a03a-96dd7a8d473e","Type":"ContainerDied","Data":"5f2ad9767a3dcf61790717cb9ef400daaf83e9a995a44d81c56d62ddb23b42cf"} Oct 31 13:55:43.199252 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.199234 5532 scope.go:122] "RemoveContainer" containerID="b9475fd85cd13f152275e658d5d399c779ab1f3e35f30dcaaaf21b6d28c718bb" Oct 31 13:55:43.200749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.200728 5532 generic.go:356] "Generic (PLEG): container finished" podID="9eef8cf1-7c9e-489c-8ec6-0b5175e28e01" containerID="9f046bd77be5df61c394c55e4747905839ecc799100720dbe8c5fe1eafddb9e9" exitCode=0 Oct 31 13:55:43.200882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.200773 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" Oct 31 13:55:43.200882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.200775 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" event={"ID":"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01","Type":"ContainerDied","Data":"9f046bd77be5df61c394c55e4747905839ecc799100720dbe8c5fe1eafddb9e9"} Oct 31 13:55:43.200966 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.200891 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk" event={"ID":"9eef8cf1-7c9e-489c-8ec6-0b5175e28e01","Type":"ContainerDied","Data":"3a45f834cf4901a809962a79391a2fbed95effbbc25efee0d581916d3ac2a806"} Oct 31 13:55:43.211328 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.211309 5532 scope.go:122] "RemoveContainer" containerID="b9475fd85cd13f152275e658d5d399c779ab1f3e35f30dcaaaf21b6d28c718bb" Oct 31 13:55:43.211723 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:55:43.211697 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9475fd85cd13f152275e658d5d399c779ab1f3e35f30dcaaaf21b6d28c718bb\": container with ID starting with b9475fd85cd13f152275e658d5d399c779ab1f3e35f30dcaaaf21b6d28c718bb not found: ID does not exist" containerID="b9475fd85cd13f152275e658d5d399c779ab1f3e35f30dcaaaf21b6d28c718bb" Oct 31 13:55:43.211838 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.211730 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9475fd85cd13f152275e658d5d399c779ab1f3e35f30dcaaaf21b6d28c718bb"} err="failed to get container status \"b9475fd85cd13f152275e658d5d399c779ab1f3e35f30dcaaaf21b6d28c718bb\": rpc error: code = NotFound desc = could not find container \"b9475fd85cd13f152275e658d5d399c779ab1f3e35f30dcaaaf21b6d28c718bb\": container with ID starting with b9475fd85cd13f152275e658d5d399c779ab1f3e35f30dcaaaf21b6d28c718bb not found: ID does not exist" Oct 31 13:55:43.211838 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.211750 5532 scope.go:122] "RemoveContainer" containerID="9f046bd77be5df61c394c55e4747905839ecc799100720dbe8c5fe1eafddb9e9" Oct 31 13:55:43.223142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.223118 5532 scope.go:122] "RemoveContainer" containerID="9f046bd77be5df61c394c55e4747905839ecc799100720dbe8c5fe1eafddb9e9" Oct 31 13:55:43.223432 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:55:43.223415 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f046bd77be5df61c394c55e4747905839ecc799100720dbe8c5fe1eafddb9e9\": container with ID starting with 9f046bd77be5df61c394c55e4747905839ecc799100720dbe8c5fe1eafddb9e9 not found: ID does not exist" containerID="9f046bd77be5df61c394c55e4747905839ecc799100720dbe8c5fe1eafddb9e9" Oct 31 13:55:43.223475 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.223440 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f046bd77be5df61c394c55e4747905839ecc799100720dbe8c5fe1eafddb9e9"} err="failed to get container status \"9f046bd77be5df61c394c55e4747905839ecc799100720dbe8c5fe1eafddb9e9\": rpc error: code = NotFound desc = could not find container \"9f046bd77be5df61c394c55e4747905839ecc799100720dbe8c5fe1eafddb9e9\": container with ID starting with 9f046bd77be5df61c394c55e4747905839ecc799100720dbe8c5fe1eafddb9e9 not found: ID does not exist" Oct 31 13:55:43.232196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.232174 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w"] Oct 31 13:55:43.239746 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.239723 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-20b8a4-575cbc6667-4wx5w"] Oct 31 13:55:43.255104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.255078 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk"] Oct 31 13:55:43.257469 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:43.257450 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-20b8a4-5d968c6d88-6rkqk"] Oct 31 13:55:44.006691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:44.006657 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3293093d-6ff2-4349-9d1d-f82bb31baf19" path="/var/lib/kubelet/pods/3293093d-6ff2-4349-9d1d-f82bb31baf19/volumes" Oct 31 13:55:44.007198 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:44.007182 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7981c832-1940-492c-a03a-96dd7a8d473e" path="/var/lib/kubelet/pods/7981c832-1940-492c-a03a-96dd7a8d473e/volumes" Oct 31 13:55:44.007628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:44.007589 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eef8cf1-7c9e-489c-8ec6-0b5175e28e01" path="/var/lib/kubelet/pods/9eef8cf1-7c9e-489c-8ec6-0b5175e28e01/volumes" Oct 31 13:55:44.846054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:44.846019 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:44.860030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:44.860004 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:45.244983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.244944 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 13:55:45.750022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.749985 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq"] Oct 31 13:55:45.751375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.751342 5532 cpu_manager.go:414] "containerMap: removing container" podUID="9eef8cf1-7c9e-489c-8ec6-0b5175e28e01" containerName="payload-processing" Oct 31 13:55:45.751375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.751371 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eef8cf1-7c9e-489c-8ec6-0b5175e28e01" containerName="payload-processing" Oct 31 13:55:45.751586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.751441 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a5335d46-2571-41b5-988e-621350b8ffac" containerName="revoke-keys" Oct 31 13:55:45.751586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.751451 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5335d46-2571-41b5-988e-621350b8ffac" containerName="revoke-keys" Oct 31 13:55:45.751586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.751505 5532 cpu_manager.go:414] "containerMap: removing container" podUID="7981c832-1940-492c-a03a-96dd7a8d473e" containerName="payload-pre-processing" Oct 31 13:55:45.751586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.751526 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7981c832-1940-492c-a03a-96dd7a8d473e" containerName="payload-pre-processing" Oct 31 13:55:45.751834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.751755 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eef8cf1-7c9e-489c-8ec6-0b5175e28e01" containerName="payload-processing" Oct 31 13:55:45.751834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.751783 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5335d46-2571-41b5-988e-621350b8ffac" containerName="revoke-keys" Oct 31 13:55:45.751834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.751801 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="7981c832-1940-492c-a03a-96dd7a8d473e" containerName="payload-pre-processing" Oct 31 13:55:45.758215 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.758190 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.769022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.768994 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq"] Oct 31 13:55:45.841467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.841430 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.841774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.841535 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.841774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.841573 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.841774 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.841763 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnjks\" (UniqueName: \"kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-kube-api-access-cnjks\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.842086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.841905 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.842086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.841964 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.842086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.841998 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.842086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.842069 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5738e7ce-0f1c-416f-b3c5-50719400f057-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.842331 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.842102 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.944081 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.944045 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnjks\" (UniqueName: \"kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-kube-api-access-cnjks\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.944282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.944149 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.944282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.944192 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.944282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.944225 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.944464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.944373 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5738e7ce-0f1c-416f-b3c5-50719400f057-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.944464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.944428 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.944581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.944537 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.944712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.944589 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.944712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.944669 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.944839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.944706 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.945037 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.945007 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-certs\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.945037 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.945028 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-credential-socket\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.945237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.945130 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-data\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.945237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.945184 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5738e7ce-0f1c-416f-b3c5-50719400f057-istiod-ca-cert\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.946670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.946651 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-envoy\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.946964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.946946 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-podinfo\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.957369 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.957344 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnjks\" (UniqueName: \"kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-kube-api-access-cnjks\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:45.957668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:45.957589 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-token\") pod \"e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:46.083504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:46.083419 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:46.383964 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:55:46.383927 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5738e7ce_0f1c_416f_b3c5_50719400f057.slice/crio-1964fa82d147667a8a1a69e5c27d0feb0b2ef270f13a6c702652f1808808fecc WatchSource:0}: Error finding container 1964fa82d147667a8a1a69e5c27d0feb0b2ef270f13a6c702652f1808808fecc: Status 404 returned error can't find the container with id 1964fa82d147667a8a1a69e5c27d0feb0b2ef270f13a6c702652f1808808fecc Oct 31 13:55:46.385642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:46.385614 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq"] Oct 31 13:55:46.387158 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:46.387124 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:46.387253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:46.387213 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:46.387253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:46.387243 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:55:47.270982 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:47.270940 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" event={"ID":"5738e7ce-0f1c-416f-b3c5-50719400f057","Type":"ContainerStarted","Data":"1cfe1acb4e20cb8eaf181878d75b86f9047143d0e81166e1de3741458626b5f4"} Oct 31 13:55:47.271196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:47.271004 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" event={"ID":"5738e7ce-0f1c-416f-b3c5-50719400f057","Type":"ContainerStarted","Data":"1964fa82d147667a8a1a69e5c27d0feb0b2ef270f13a6c702652f1808808fecc"} Oct 31 13:55:47.299796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:47.299749 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" podStartSLOduration=2.299735057 podStartE2EDuration="2.299735057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:47.299527068 +0000 UTC m=+3115.445270896" watchObservedRunningTime="2026-10-31 13:55:47.299735057 +0000 UTC m=+3115.445478885" Oct 31 13:55:48.084216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:48.084187 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:48.086453 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:48.086427 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 31 13:55:48.086606 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:48.086485 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" podUID="5738e7ce-0f1c-416f-b3c5-50719400f057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 31 13:55:49.084488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:49.084448 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 31 13:55:49.085034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:49.084503 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" podUID="5738e7ce-0f1c-416f-b3c5-50719400f057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 31 13:55:50.084422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:50.084387 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 31 13:55:50.084643 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:50.084451 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" podUID="5738e7ce-0f1c-416f-b3c5-50719400f057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 31 13:55:51.085950 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.085915 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 31 13:55:51.086498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.086457 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" podUID="5738e7ce-0f1c-416f-b3c5-50719400f057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 31 13:55:51.327845 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.327810 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj"] Oct 31 13:55:51.333170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.333148 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" Oct 31 13:55:51.379091 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.379059 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj"] Oct 31 13:55:51.393279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.393233 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:55:51.399070 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.399039 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:55:51.399201 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.399124 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:55:51.414555 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.414529 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28mpp\" (UniqueName: \"kubernetes.io/projected/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-kube-api-access-28mpp\") pod \"maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj\" (UID: \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" Oct 31 13:55:51.414693 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.414609 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj\" (UID: \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" Oct 31 13:55:51.414837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.414817 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj\" (UID: \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" Oct 31 13:55:51.542619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.541102 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj\" (UID: \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" Oct 31 13:55:51.542619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.541301 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28mpp\" (UniqueName: \"kubernetes.io/projected/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-kube-api-access-28mpp\") pod \"maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj\" (UID: \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" Oct 31 13:55:51.542619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.541378 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj\" (UID: \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" Oct 31 13:55:51.558616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.553468 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj\" (UID: \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" Oct 31 13:55:51.596261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.596226 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28mpp\" (UniqueName: \"kubernetes.io/projected/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-kube-api-access-28mpp\") pod \"maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj\" (UID: \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" Oct 31 13:55:51.616317 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.616288 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj\" (UID: \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" Oct 31 13:55:51.679956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:51.679915 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" Oct 31 13:55:51.999558 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:55:51.999529 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0526eecc_8f8c_4ac5_9a54_601cc0e6f692.slice/crio-7302f73a301deaf0762f6590bb558c3f65e550b407e1fc4987295e4b9ff55043 WatchSource:0}: Error finding container 7302f73a301deaf0762f6590bb558c3f65e550b407e1fc4987295e4b9ff55043: Status 404 returned error can't find the container with id 7302f73a301deaf0762f6590bb558c3f65e550b407e1fc4987295e4b9ff55043 Oct 31 13:55:52.010206 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:52.010177 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj"] Oct 31 13:55:52.084330 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:52.084296 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" start-of-body= Oct 31 13:55:52.084468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:52.084357 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" podUID="5738e7ce-0f1c-416f-b3c5-50719400f057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.13:15021/healthz/ready\": dial tcp 10.132.1.13:15021: connect: connection refused" Oct 31 13:55:52.423667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:52.423633 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9"] Oct 31 13:55:52.566093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:52.566060 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" event={"ID":"0526eecc-8f8c-4ac5-9a54-601cc0e6f692","Type":"ContainerStarted","Data":"b9cb75f4ce6ececaf3641da8b00e079c6475cee761e875908be40b18f5a5dc84"} Oct 31 13:55:52.566235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:52.566127 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" event={"ID":"0526eecc-8f8c-4ac5-9a54-601cc0e6f692","Type":"ContainerStarted","Data":"7302f73a301deaf0762f6590bb558c3f65e550b407e1fc4987295e4b9ff55043"} Oct 31 13:55:52.590181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:52.590113 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" podStartSLOduration=1.590093344 podStartE2EDuration="1.590093344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:55:52.586166596 +0000 UTC m=+3120.731910486" watchObservedRunningTime="2026-10-31 13:55:52.590093344 +0000 UTC m=+3120.735837172" Oct 31 13:55:53.089376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:53.089347 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:53.089800 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:53.089770 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:53.090425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:53.090400 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 13:55:53.630639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:53.630139 5532 generic.go:356] "Generic (PLEG): container finished" podID="0526eecc-8f8c-4ac5-9a54-601cc0e6f692" containerID="b9cb75f4ce6ececaf3641da8b00e079c6475cee761e875908be40b18f5a5dc84" exitCode=0 Oct 31 13:55:53.632941 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:53.631737 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" event={"ID":"0526eecc-8f8c-4ac5-9a54-601cc0e6f692","Type":"ContainerDied","Data":"b9cb75f4ce6ececaf3641da8b00e079c6475cee761e875908be40b18f5a5dc84"} Oct 31 13:55:54.666234 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.666196 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" podUID="2257c6d8-8a08-4d03-9bad-48dd3493aecd" containerName="main" containerID="cri-o://bb5322994802242b1b5e2f833cce8d3d817fc084a04541331b5d89acf7c8a3c1" gracePeriod=45 Oct 31 13:55:54.828453 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.828422 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" Oct 31 13:55:54.876396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.876367 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-xks-ca\") pod \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\" (UID: \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\") " Oct 31 13:55:54.876607 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.876417 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-openshift-ca\") pod \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\" (UID: \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\") " Oct 31 13:55:54.876607 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.876498 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-kube-api-access-28mpp\" (UniqueName: \"kubernetes.io/projected/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-kube-api-access-28mpp\") pod \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\" (UID: \"0526eecc-8f8c-4ac5-9a54-601cc0e6f692\") " Oct 31 13:55:54.877245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.877211 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-openshift-ca" pod "0526eecc-8f8c-4ac5-9a54-601cc0e6f692" (UID: "0526eecc-8f8c-4ac5-9a54-601cc0e6f692"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:54.879685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.879653 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-kube-api-access-28mpp" pod "0526eecc-8f8c-4ac5-9a54-601cc0e6f692" (UID: "0526eecc-8f8c-4ac5-9a54-601cc0e6f692"). InnerVolumeSpecName "kube-api-access-28mpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:54.879814 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.879659 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-xks-ca" pod "0526eecc-8f8c-4ac5-9a54-601cc0e6f692" (UID: "0526eecc-8f8c-4ac5-9a54-601cc0e6f692"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:55:54.930699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.930671 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:55:54.978228 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.978201 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tls-certs\" (UniqueName: \"kubernetes.io/secret/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tls-certs\") pod \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " Oct 31 13:55:54.978424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.978236 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-dshm\") pod \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " Oct 31 13:55:54.978424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.978401 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-model-cache\") pod \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " Oct 31 13:55:54.978562 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.978534 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-home\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-home\") pod \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " Oct 31 13:55:54.978647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.978580 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2257c6d8-8a08-4d03-9bad-48dd3493aecd-kube-api-access-qxz9f\" (UniqueName: \"kubernetes.io/projected/2257c6d8-8a08-4d03-9bad-48dd3493aecd-kube-api-access-qxz9f\") pod \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " Oct 31 13:55:54.978722 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.978677 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tmp-dir\") pod \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\" (UID: \"2257c6d8-8a08-4d03-9bad-48dd3493aecd\") " Oct 31 13:55:54.978884 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.978855 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-home" pod "2257c6d8-8a08-4d03-9bad-48dd3493aecd" (UID: "2257c6d8-8a08-4d03-9bad-48dd3493aecd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:55:54.979093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.979072 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tmp-dir" pod "2257c6d8-8a08-4d03-9bad-48dd3493aecd" (UID: "2257c6d8-8a08-4d03-9bad-48dd3493aecd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:55:54.979677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.979649 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28mpp\" (UniqueName: \"kubernetes.io/projected/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-kube-api-access-28mpp\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:54.979783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.979681 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:54.979783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.979697 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:54.979783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.979712 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0526eecc-8f8c-4ac5-9a54-601cc0e6f692-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:54.979783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.979727 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:54.980093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.980070 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-model-cache" pod "2257c6d8-8a08-4d03-9bad-48dd3493aecd" (UID: "2257c6d8-8a08-4d03-9bad-48dd3493aecd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:55:54.981240 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.981215 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-dshm" pod "2257c6d8-8a08-4d03-9bad-48dd3493aecd" (UID: "2257c6d8-8a08-4d03-9bad-48dd3493aecd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:55:54.981325 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.981268 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2257c6d8-8a08-4d03-9bad-48dd3493aecd-kube-api-access-qxz9f" pod "2257c6d8-8a08-4d03-9bad-48dd3493aecd" (UID: "2257c6d8-8a08-4d03-9bad-48dd3493aecd"). InnerVolumeSpecName "kube-api-access-qxz9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:54.981861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:54.981835 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tls-certs" pod "2257c6d8-8a08-4d03-9bad-48dd3493aecd" (UID: "2257c6d8-8a08-4d03-9bad-48dd3493aecd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:55:55.081372 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.081342 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxz9f\" (UniqueName: \"kubernetes.io/projected/2257c6d8-8a08-4d03-9bad-48dd3493aecd-kube-api-access-qxz9f\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:55.081372 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.081371 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2257c6d8-8a08-4d03-9bad-48dd3493aecd-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:55.081585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.081385 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:55.081585 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.081398 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2257c6d8-8a08-4d03-9bad-48dd3493aecd-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:55.649121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.649094 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" Oct 31 13:55:55.649307 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.649129 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj" event={"ID":"0526eecc-8f8c-4ac5-9a54-601cc0e6f692","Type":"ContainerDied","Data":"7302f73a301deaf0762f6590bb558c3f65e550b407e1fc4987295e4b9ff55043"} Oct 31 13:55:55.649307 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.649182 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7302f73a301deaf0762f6590bb558c3f65e550b407e1fc4987295e4b9ff55043" Oct 31 13:55:55.650973 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.650948 5532 generic.go:356] "Generic (PLEG): container finished" podID="2257c6d8-8a08-4d03-9bad-48dd3493aecd" containerID="bb5322994802242b1b5e2f833cce8d3d817fc084a04541331b5d89acf7c8a3c1" exitCode=0 Oct 31 13:55:55.651085 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.650992 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" Oct 31 13:55:55.651085 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.651003 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" event={"ID":"2257c6d8-8a08-4d03-9bad-48dd3493aecd","Type":"ContainerDied","Data":"bb5322994802242b1b5e2f833cce8d3d817fc084a04541331b5d89acf7c8a3c1"} Oct 31 13:55:55.651085 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.651042 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw" event={"ID":"2257c6d8-8a08-4d03-9bad-48dd3493aecd","Type":"ContainerDied","Data":"ea7f9077afdd8eba8474af3ff0f8dc1efb62cf2d276de0c68e2aa50613f46ace"} Oct 31 13:55:55.651085 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.651058 5532 scope.go:122] "RemoveContainer" containerID="bb5322994802242b1b5e2f833cce8d3d817fc084a04541331b5d89acf7c8a3c1" Oct 31 13:55:55.662984 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.662966 5532 scope.go:122] "RemoveContainer" containerID="bb5322994802242b1b5e2f833cce8d3d817fc084a04541331b5d89acf7c8a3c1" Oct 31 13:55:55.663232 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:55:55.663212 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb5322994802242b1b5e2f833cce8d3d817fc084a04541331b5d89acf7c8a3c1\": container with ID starting with bb5322994802242b1b5e2f833cce8d3d817fc084a04541331b5d89acf7c8a3c1 not found: ID does not exist" containerID="bb5322994802242b1b5e2f833cce8d3d817fc084a04541331b5d89acf7c8a3c1" Oct 31 13:55:55.663280 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.663237 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb5322994802242b1b5e2f833cce8d3d817fc084a04541331b5d89acf7c8a3c1"} err="failed to get container status \"bb5322994802242b1b5e2f833cce8d3d817fc084a04541331b5d89acf7c8a3c1\": rpc error: code = NotFound desc = could not find container \"bb5322994802242b1b5e2f833cce8d3d817fc084a04541331b5d89acf7c8a3c1\": container with ID starting with bb5322994802242b1b5e2f833cce8d3d817fc084a04541331b5d89acf7c8a3c1 not found: ID does not exist" Oct 31 13:55:55.686347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.686320 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw"] Oct 31 13:55:55.691246 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.691224 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-14ba5a/auth-test-model-14ba5a-kserve-7f7fbfc469-ks2dw"] Oct 31 13:55:55.830989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.830961 5532 scope.go:122] "RemoveContainer" containerID="4a3dcfe9b694f73e55d5eff26c0a59054f5eba4e7366ef7a620f67baf0937787" Oct 31 13:55:55.842436 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.842418 5532 scope.go:122] "RemoveContainer" containerID="2641a177e6570c7ed8fc82e0436ac3cb1942c627a4af2b1f76df158e9e45ea89" Oct 31 13:55:55.853434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:55.853414 5532 scope.go:122] "RemoveContainer" containerID="5f60c1e137fa099b942ed63c7e63ec245d002225a91c58c65af6961904871147" Oct 31 13:55:56.004993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:56.004959 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2257c6d8-8a08-4d03-9bad-48dd3493aecd" path="/var/lib/kubelet/pods/2257c6d8-8a08-4d03-9bad-48dd3493aecd/volumes" Oct 31 13:55:57.424833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:57.424762 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" podUID="231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3" containerName="payload-processing" containerID="cri-o://a8d9da76db4530ac27c7fcf8fd9d66e58c61361e34d2e0e81e8fef3f89782d2d" gracePeriod=25 Oct 31 13:55:57.674193 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:57.674157 5532 generic.go:356] "Generic (PLEG): container finished" podID="231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3" containerID="a8d9da76db4530ac27c7fcf8fd9d66e58c61361e34d2e0e81e8fef3f89782d2d" exitCode=0 Oct 31 13:55:57.674399 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:57.674232 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" event={"ID":"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3","Type":"ContainerDied","Data":"a8d9da76db4530ac27c7fcf8fd9d66e58c61361e34d2e0e81e8fef3f89782d2d"} Oct 31 13:55:57.722406 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:57.722330 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch"] Oct 31 13:55:57.729777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:57.729747 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z"] Oct 31 13:55:57.774748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:57.774721 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" Oct 31 13:55:57.829911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:57.829875 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-plugins-config-volume\") pod \"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3\" (UID: \"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3\") " Oct 31 13:55:57.830109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:57.830057 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-kube-api-access-rnnvx\" (UniqueName: \"kubernetes.io/projected/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-kube-api-access-rnnvx\") pod \"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3\" (UID: \"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3\") " Oct 31 13:55:57.830339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:57.830316 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-plugins-config-volume" pod "231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3" (UID: "231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:55:57.830769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:57.830740 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:57.832501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:57.832463 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-kube-api-access-rnnvx" pod "231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3" (UID: "231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3"). InnerVolumeSpecName "kube-api-access-rnnvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:55:57.932324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:57.932287 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnnvx\" (UniqueName: \"kubernetes.io/projected/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3-kube-api-access-rnnvx\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:55:58.684711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:58.684680 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" Oct 31 13:55:58.684711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:58.684705 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9" event={"ID":"231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3","Type":"ContainerDied","Data":"17c0416dff1bf56fc2713705a4ceb34fc6aae0627c7af9be050ed9eaab510551"} Oct 31 13:55:58.685235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:58.684766 5532 scope.go:122] "RemoveContainer" containerID="a8d9da76db4530ac27c7fcf8fd9d66e58c61361e34d2e0e81e8fef3f89782d2d" Oct 31 13:55:58.708971 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:58.708945 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9"] Oct 31 13:55:58.715564 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:58.715537 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dafbce-7f4cf56cc7-dgnx9"] Oct 31 13:55:59.048379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.048306 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp"] Oct 31 13:55:59.049163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.049146 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2257c6d8-8a08-4d03-9bad-48dd3493aecd" containerName="main" Oct 31 13:55:59.049220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.049164 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2257c6d8-8a08-4d03-9bad-48dd3493aecd" containerName="main" Oct 31 13:55:59.049220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.049178 5532 cpu_manager.go:414] "containerMap: removing container" podUID="0526eecc-8f8c-4ac5-9a54-601cc0e6f692" containerName="revoke-keys" Oct 31 13:55:59.049220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.049185 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0526eecc-8f8c-4ac5-9a54-601cc0e6f692" containerName="revoke-keys" Oct 31 13:55:59.049220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.049207 5532 cpu_manager.go:414] "containerMap: removing container" podUID="231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3" containerName="payload-processing" Oct 31 13:55:59.049220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.049213 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3" containerName="payload-processing" Oct 31 13:55:59.049383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.049340 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2257c6d8-8a08-4d03-9bad-48dd3493aecd" containerName="main" Oct 31 13:55:59.049383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.049351 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="0526eecc-8f8c-4ac5-9a54-601cc0e6f692" containerName="revoke-keys" Oct 31 13:55:59.049383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.049361 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3" containerName="payload-processing" Oct 31 13:55:59.054682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.054662 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.062332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.062306 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp"] Oct 31 13:55:59.062568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.062550 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-974711\"/\"default-dockercfg-4szqx\"" Oct 31 13:55:59.062907 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.062878 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-974711\"/\"rate-test-model-974711-kserve-self-signed-certs\"" Oct 31 13:55:59.063527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.063508 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-974711\"/\"kube-root-ca.crt\"" Oct 31 13:55:59.063779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.063763 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-974711\"/\"openshift-service-ca.crt\"" Oct 31 13:55:59.161045 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.161014 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-home\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.161217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.161058 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tls-certs\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.161217 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.161194 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-dshm\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.161358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.161343 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjhmx\" (UniqueName: \"kubernetes.io/projected/124cc2c6-45cd-4bcd-9714-d4041f054bbf-kube-api-access-mjhmx\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.161466 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.161453 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-model-cache\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.161618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.161586 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tmp-dir\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.263282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.263247 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjhmx\" (UniqueName: \"kubernetes.io/projected/124cc2c6-45cd-4bcd-9714-d4041f054bbf-kube-api-access-mjhmx\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.263449 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.263403 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-model-cache\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.263528 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.263514 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tmp-dir\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.263645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.263546 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-home\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.263909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.263587 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tls-certs\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.264535 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.263950 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tmp-dir\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.264715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.264545 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-dshm\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.264819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.264173 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-model-cache\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.264973 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.263982 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-home\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.267788 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.267765 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-dshm\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.268176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.268153 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tls-certs\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.278428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.278401 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjhmx\" (UniqueName: \"kubernetes.io/projected/124cc2c6-45cd-4bcd-9714-d4041f054bbf-kube-api-access-mjhmx\") pod \"rate-test-model-974711-kserve-6b6476cfdf-6bnxp\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.372305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.371925 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:55:59.671645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.671463 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp"] Oct 31 13:55:59.671759 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:55:59.671711 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod124cc2c6_45cd_4bcd_9714_d4041f054bbf.slice/crio-ec31a9c57b216d5f7ef4f14c1b8d810f76e349a2770938e98bd099cb8792d210 WatchSource:0}: Error finding container ec31a9c57b216d5f7ef4f14c1b8d810f76e349a2770938e98bd099cb8792d210: Status 404 returned error can't find the container with id ec31a9c57b216d5f7ef4f14c1b8d810f76e349a2770938e98bd099cb8792d210 Oct 31 13:55:59.697751 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:55:59.697716 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" event={"ID":"124cc2c6-45cd-4bcd-9714-d4041f054bbf","Type":"ContainerStarted","Data":"ec31a9c57b216d5f7ef4f14c1b8d810f76e349a2770938e98bd099cb8792d210"} Oct 31 13:56:00.008550 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:00.008465 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3" path="/var/lib/kubelet/pods/231aefe2-c7a4-4b5d-9b9f-fdf6aa49e5b3/volumes" Oct 31 13:56:00.032250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:00.032215 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6"] Oct 31 13:56:00.052417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:00.052376 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516f84cd-x9lr6"] Oct 31 13:56:00.708661 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:00.708624 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" event={"ID":"124cc2c6-45cd-4bcd-9714-d4041f054bbf","Type":"ContainerStarted","Data":"7c53d4593bbb0b4a0f18e24e4d7a5a5d2524a83a2ce04d88ded53d159a3eb78b"} Oct 31 13:56:00.736833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:00.736776 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" podStartSLOduration=1.7367622169999999 podStartE2EDuration="1.736762217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:00.734566813 +0000 UTC m=+3128.880310642" watchObservedRunningTime="2026-10-31 13:56:00.736762217 +0000 UTC m=+3128.882506054" Oct 31 13:56:01.372357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.372320 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:56:01.675334 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.675303 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4"] Oct 31 13:56:01.680580 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.680557 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.686867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.686839 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4"] Oct 31 13:56:01.687029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.687006 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-35d13e\"/\"kube-root-ca.crt\"" Oct 31 13:56:01.688060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.688027 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-35d13e\"/\"openshift-service-ca.crt\"" Oct 31 13:56:01.688060 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.688057 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-35d13e\"/\"default-dockercfg-fnmmt\"" Oct 31 13:56:01.688223 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.688120 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-35d13e\"/\"sub-test-model-35d13e-kserve-self-signed-certs\"" Oct 31 13:56:01.763241 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:56:01.763204 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:56:01.810644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.810589 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-model-cache\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.810810 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.810676 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tmp-dir\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.810852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.810823 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-home\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.811018 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.811002 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqvk4\" (UniqueName: \"kubernetes.io/projected/61795f0c-b2b7-46a8-95b8-5f132f863cf7-kube-api-access-qqvk4\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.811061 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.811031 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-dshm\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.811241 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.811221 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tls-certs\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.920670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.912773 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tls-certs\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.920670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.912840 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-model-cache\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.920670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.912871 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tmp-dir\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.920670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.913004 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-home\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.920670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.913180 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqvk4\" (UniqueName: \"kubernetes.io/projected/61795f0c-b2b7-46a8-95b8-5f132f863cf7-kube-api-access-qqvk4\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.920670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.913212 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-dshm\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.920670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.917980 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tmp-dir\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.920670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.918023 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-home\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.920670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.918224 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-model-cache\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.923687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.923658 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-dshm\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.931540 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.931463 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tls-certs\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.936893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.936860 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqvk4\" (UniqueName: \"kubernetes.io/projected/61795f0c-b2b7-46a8-95b8-5f132f863cf7-kube-api-access-qqvk4\") pod \"sub-test-model-35d13e-kserve-58587ffff5-8s9k4\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:01.997826 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:01.997787 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:02.013576 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:02.013537 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2" path="/var/lib/kubelet/pods/832fb8fd-1d03-46ee-bdb5-4bcaba74e1b2/volumes" Oct 31 13:56:02.336792 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:56:02.336759 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61795f0c_b2b7_46a8_95b8_5f132f863cf7.slice/crio-c707a9135b028bf9dfbbb23ee3c1a9fc5ccb651283640e2bf6b2d99626e0e8b2 WatchSource:0}: Error finding container c707a9135b028bf9dfbbb23ee3c1a9fc5ccb651283640e2bf6b2d99626e0e8b2: Status 404 returned error can't find the container with id c707a9135b028bf9dfbbb23ee3c1a9fc5ccb651283640e2bf6b2d99626e0e8b2 Oct 31 13:56:02.337079 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:02.337061 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4"] Oct 31 13:56:02.723365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:02.723332 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" podUID="9445ef10-8988-4614-80e0-7eb52d5143e1" containerName="payload-processing" containerID="cri-o://ea925a42699a72c01d80de5d38314ceab42b8dd11c7b0f34f6044b2804374a06" gracePeriod=25 Oct 31 13:56:02.728013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:02.727983 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" event={"ID":"61795f0c-b2b7-46a8-95b8-5f132f863cf7","Type":"ContainerStarted","Data":"75e5a16d934222f4eede96b40d988dfc26226b9dd414bc09829989b84cc6019b"} Oct 31 13:56:02.728013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:02.728032 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" event={"ID":"61795f0c-b2b7-46a8-95b8-5f132f863cf7","Type":"ContainerStarted","Data":"c707a9135b028bf9dfbbb23ee3c1a9fc5ccb651283640e2bf6b2d99626e0e8b2"} Oct 31 13:56:02.730509 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:02.730479 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" podUID="96216deb-419a-4c46-8132-fb54d26929eb" containerName="payload-pre-processing" containerID="cri-o://9f297f1685947ff8dc4b623858e66253c35cef0c0cbbe293b551a5e883397f49" gracePeriod=25 Oct 31 13:56:02.752268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:02.752218 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" podStartSLOduration=1.7522043630000002 podStartE2EDuration="1.752204363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:02.749982186 +0000 UTC m=+3130.895726014" watchObservedRunningTime="2026-10-31 13:56:02.752204363 +0000 UTC m=+3130.897948191" Oct 31 13:56:02.826474 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:56:02.826446 5532 logging.go:55] [core] [Channel #9523 SubChannel #9524]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.4:50052", ServerName: "10.132.1.4:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.4:50052: connect: connection refused" Oct 31 13:56:02.998137 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:02.998065 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:03.091765 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.091717 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" Oct 31 13:56:03.158842 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.158811 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" Oct 31 13:56:03.233993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.233949 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96216deb-419a-4c46-8132-fb54d26929eb-kube-api-access-k2hrd\" (UniqueName: \"kubernetes.io/projected/96216deb-419a-4c46-8132-fb54d26929eb-kube-api-access-k2hrd\") pod \"96216deb-419a-4c46-8132-fb54d26929eb\" (UID: \"96216deb-419a-4c46-8132-fb54d26929eb\") " Oct 31 13:56:03.234189 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.234165 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96216deb-419a-4c46-8132-fb54d26929eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96216deb-419a-4c46-8132-fb54d26929eb-plugins-config-volume\") pod \"96216deb-419a-4c46-8132-fb54d26929eb\" (UID: \"96216deb-419a-4c46-8132-fb54d26929eb\") " Oct 31 13:56:03.234455 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.234433 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9445ef10-8988-4614-80e0-7eb52d5143e1-kube-api-access-hdnqb\" (UniqueName: \"kubernetes.io/projected/9445ef10-8988-4614-80e0-7eb52d5143e1-kube-api-access-hdnqb\") pod \"9445ef10-8988-4614-80e0-7eb52d5143e1\" (UID: \"9445ef10-8988-4614-80e0-7eb52d5143e1\") " Oct 31 13:56:03.235001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.234482 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9445ef10-8988-4614-80e0-7eb52d5143e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9445ef10-8988-4614-80e0-7eb52d5143e1-plugins-config-volume\") pod \"9445ef10-8988-4614-80e0-7eb52d5143e1\" (UID: \"9445ef10-8988-4614-80e0-7eb52d5143e1\") " Oct 31 13:56:03.235001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.234581 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96216deb-419a-4c46-8132-fb54d26929eb-plugins-config-volume" pod "96216deb-419a-4c46-8132-fb54d26929eb" (UID: "96216deb-419a-4c46-8132-fb54d26929eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:03.235001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.234982 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9445ef10-8988-4614-80e0-7eb52d5143e1-plugins-config-volume" pod "9445ef10-8988-4614-80e0-7eb52d5143e1" (UID: "9445ef10-8988-4614-80e0-7eb52d5143e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:03.235233 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.235207 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9445ef10-8988-4614-80e0-7eb52d5143e1-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:03.235296 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.235252 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96216deb-419a-4c46-8132-fb54d26929eb-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:03.236277 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.236247 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96216deb-419a-4c46-8132-fb54d26929eb-kube-api-access-k2hrd" pod "96216deb-419a-4c46-8132-fb54d26929eb" (UID: "96216deb-419a-4c46-8132-fb54d26929eb"). InnerVolumeSpecName "kube-api-access-k2hrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:03.237253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.237226 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9445ef10-8988-4614-80e0-7eb52d5143e1-kube-api-access-hdnqb" pod "9445ef10-8988-4614-80e0-7eb52d5143e1" (UID: "9445ef10-8988-4614-80e0-7eb52d5143e1"). InnerVolumeSpecName "kube-api-access-hdnqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:03.336955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.336883 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdnqb\" (UniqueName: \"kubernetes.io/projected/9445ef10-8988-4614-80e0-7eb52d5143e1-kube-api-access-hdnqb\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:03.336955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.336909 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2hrd\" (UniqueName: \"kubernetes.io/projected/96216deb-419a-4c46-8132-fb54d26929eb-kube-api-access-k2hrd\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:03.738413 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.738367 5532 generic.go:356] "Generic (PLEG): container finished" podID="9445ef10-8988-4614-80e0-7eb52d5143e1" containerID="ea925a42699a72c01d80de5d38314ceab42b8dd11c7b0f34f6044b2804374a06" exitCode=0 Oct 31 13:56:03.738649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.738423 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" Oct 31 13:56:03.738649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.738442 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" event={"ID":"9445ef10-8988-4614-80e0-7eb52d5143e1","Type":"ContainerDied","Data":"ea925a42699a72c01d80de5d38314ceab42b8dd11c7b0f34f6044b2804374a06"} Oct 31 13:56:03.738649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.738527 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch" event={"ID":"9445ef10-8988-4614-80e0-7eb52d5143e1","Type":"ContainerDied","Data":"ccfb20eba8307ac58d170123a02e2e1999812bbfe249a218c91854c7498d6224"} Oct 31 13:56:03.738649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.738551 5532 scope.go:122] "RemoveContainer" containerID="ea925a42699a72c01d80de5d38314ceab42b8dd11c7b0f34f6044b2804374a06" Oct 31 13:56:03.740197 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.740170 5532 generic.go:356] "Generic (PLEG): container finished" podID="96216deb-419a-4c46-8132-fb54d26929eb" containerID="9f297f1685947ff8dc4b623858e66253c35cef0c0cbbe293b551a5e883397f49" exitCode=0 Oct 31 13:56:03.740384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.740236 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" Oct 31 13:56:03.740384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.740246 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" event={"ID":"96216deb-419a-4c46-8132-fb54d26929eb","Type":"ContainerDied","Data":"9f297f1685947ff8dc4b623858e66253c35cef0c0cbbe293b551a5e883397f49"} Oct 31 13:56:03.740384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.740304 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" event={"ID":"96216deb-419a-4c46-8132-fb54d26929eb","Type":"ContainerDied","Data":"284c36f2bd58db6a0e42e4e05995f46c4ddd177629403c71d80704d477264ea3"} Oct 31 13:56:03.754236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.754220 5532 scope.go:122] "RemoveContainer" containerID="ea925a42699a72c01d80de5d38314ceab42b8dd11c7b0f34f6044b2804374a06" Oct 31 13:56:03.754532 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:56:03.754513 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea925a42699a72c01d80de5d38314ceab42b8dd11c7b0f34f6044b2804374a06\": container with ID starting with ea925a42699a72c01d80de5d38314ceab42b8dd11c7b0f34f6044b2804374a06 not found: ID does not exist" containerID="ea925a42699a72c01d80de5d38314ceab42b8dd11c7b0f34f6044b2804374a06" Oct 31 13:56:03.754576 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.754540 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea925a42699a72c01d80de5d38314ceab42b8dd11c7b0f34f6044b2804374a06"} err="failed to get container status \"ea925a42699a72c01d80de5d38314ceab42b8dd11c7b0f34f6044b2804374a06\": rpc error: code = NotFound desc = could not find container \"ea925a42699a72c01d80de5d38314ceab42b8dd11c7b0f34f6044b2804374a06\": container with ID starting with ea925a42699a72c01d80de5d38314ceab42b8dd11c7b0f34f6044b2804374a06 not found: ID does not exist" Oct 31 13:56:03.754576 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.754557 5532 scope.go:122] "RemoveContainer" containerID="9f297f1685947ff8dc4b623858e66253c35cef0c0cbbe293b551a5e883397f49" Oct 31 13:56:03.765716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.765699 5532 scope.go:122] "RemoveContainer" containerID="9f297f1685947ff8dc4b623858e66253c35cef0c0cbbe293b551a5e883397f49" Oct 31 13:56:03.766019 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:56:03.765994 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f297f1685947ff8dc4b623858e66253c35cef0c0cbbe293b551a5e883397f49\": container with ID starting with 9f297f1685947ff8dc4b623858e66253c35cef0c0cbbe293b551a5e883397f49 not found: ID does not exist" containerID="9f297f1685947ff8dc4b623858e66253c35cef0c0cbbe293b551a5e883397f49" Oct 31 13:56:03.766074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.766019 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f297f1685947ff8dc4b623858e66253c35cef0c0cbbe293b551a5e883397f49"} err="failed to get container status \"9f297f1685947ff8dc4b623858e66253c35cef0c0cbbe293b551a5e883397f49\": rpc error: code = NotFound desc = could not find container \"9f297f1685947ff8dc4b623858e66253c35cef0c0cbbe293b551a5e883397f49\": container with ID starting with 9f297f1685947ff8dc4b623858e66253c35cef0c0cbbe293b551a5e883397f49 not found: ID does not exist" Oct 31 13:56:03.769069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.769047 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch"] Oct 31 13:56:03.773883 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.773861 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-403dff-74c67b65b9-smzch"] Oct 31 13:56:03.788197 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.788171 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z"] Oct 31 13:56:03.793261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.793236 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z"] Oct 31 13:56:03.826991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:03.826963 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-403dff-57764cfb7-vld2z" podUID="96216deb-419a-4c46-8132-fb54d26929eb" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.4:50052\" within 1s: context deadline exceeded" Oct 31 13:56:04.006207 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:04.006135 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9445ef10-8988-4614-80e0-7eb52d5143e1" path="/var/lib/kubelet/pods/9445ef10-8988-4614-80e0-7eb52d5143e1/volumes" Oct 31 13:56:04.006668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:04.006653 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96216deb-419a-4c46-8132-fb54d26929eb" path="/var/lib/kubelet/pods/96216deb-419a-4c46-8132-fb54d26929eb/volumes" Oct 31 13:56:07.521460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:07.521424 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp"] Oct 31 13:56:07.522057 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:07.521659 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" podUID="5e7f54e9-9441-4310-8129-8f7bf99f3bc6" containerName="maas-api" containerID="cri-o://5d64728a49adb48d5a5a4cd7037a2f982f826359dec231654ae1bb48edc9949d" gracePeriod=30 Oct 31 13:56:07.614139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:07.614106 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv"] Oct 31 13:56:07.614398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:07.614370 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" podUID="b866dadc-bcaf-4a67-8edd-beada3347fe5" containerName="payload-processing" containerID="cri-o://6d5ca2b5d2bbb512e4838aaa2a29ebdd5fccc9de85efbf57aa73f0997acff1dd" gracePeriod=30 Oct 31 13:56:07.624520 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:07.624487 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj"] Oct 31 13:56:07.624769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:07.624700 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" podUID="92671137-3b3e-40d4-bb9d-48eabf91adb3" containerName="payload-pre-processing" containerID="cri-o://22f6bc4e03f2067fcdedbec64254855c0f2b450c166f2e295322ffffd35dbaf3" gracePeriod=30 Oct 31 13:56:07.790700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:07.789395 5532 generic.go:356] "Generic (PLEG): container finished" podID="b866dadc-bcaf-4a67-8edd-beada3347fe5" containerID="6d5ca2b5d2bbb512e4838aaa2a29ebdd5fccc9de85efbf57aa73f0997acff1dd" exitCode=0 Oct 31 13:56:07.790700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:07.789625 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" event={"ID":"b866dadc-bcaf-4a67-8edd-beada3347fe5","Type":"ContainerDied","Data":"6d5ca2b5d2bbb512e4838aaa2a29ebdd5fccc9de85efbf57aa73f0997acff1dd"} Oct 31 13:56:07.792985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:07.792957 5532 generic.go:356] "Generic (PLEG): container finished" podID="5e7f54e9-9441-4310-8129-8f7bf99f3bc6" containerID="5d64728a49adb48d5a5a4cd7037a2f982f826359dec231654ae1bb48edc9949d" exitCode=0 Oct 31 13:56:07.793133 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:07.793066 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" event={"ID":"5e7f54e9-9441-4310-8129-8f7bf99f3bc6","Type":"ContainerDied","Data":"5d64728a49adb48d5a5a4cd7037a2f982f826359dec231654ae1bb48edc9949d"} Oct 31 13:56:07.795624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:07.794755 5532 generic.go:356] "Generic (PLEG): container finished" podID="92671137-3b3e-40d4-bb9d-48eabf91adb3" containerID="22f6bc4e03f2067fcdedbec64254855c0f2b450c166f2e295322ffffd35dbaf3" exitCode=0 Oct 31 13:56:07.795624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:07.794793 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" event={"ID":"92671137-3b3e-40d4-bb9d-48eabf91adb3","Type":"ContainerDied","Data":"22f6bc4e03f2067fcdedbec64254855c0f2b450c166f2e295322ffffd35dbaf3"} Oct 31 13:56:08.032891 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.032858 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:56:08.063165 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.063067 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-openshift-service-ca\") pod \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " Oct 31 13:56:08.063549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.063523 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-kube-api-access-227pw\" (UniqueName: \"kubernetes.io/projected/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-kube-api-access-227pw\") pod \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " Oct 31 13:56:08.063820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.063800 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-metrics-certs\") pod \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " Oct 31 13:56:08.064064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.064048 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-maas-api-tls\") pod \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\" (UID: \"5e7f54e9-9441-4310-8129-8f7bf99f3bc6\") " Oct 31 13:56:08.068842 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.068805 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-openshift-service-ca" pod "5e7f54e9-9441-4310-8129-8f7bf99f3bc6" (UID: "5e7f54e9-9441-4310-8129-8f7bf99f3bc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:08.073339 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.072905 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-metrics-certs" pod "5e7f54e9-9441-4310-8129-8f7bf99f3bc6" (UID: "5e7f54e9-9441-4310-8129-8f7bf99f3bc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:56:08.077902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.076673 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-maas-api-tls" pod "5e7f54e9-9441-4310-8129-8f7bf99f3bc6" (UID: "5e7f54e9-9441-4310-8129-8f7bf99f3bc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:56:08.078172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.078139 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-kube-api-access-227pw" pod "5e7f54e9-9441-4310-8129-8f7bf99f3bc6" (UID: "5e7f54e9-9441-4310-8129-8f7bf99f3bc6"). InnerVolumeSpecName "kube-api-access-227pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:08.139486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.139462 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" Oct 31 13:56:08.166028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.165996 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b866dadc-bcaf-4a67-8edd-beada3347fe5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b866dadc-bcaf-4a67-8edd-beada3347fe5-plugins-config-volume\") pod \"b866dadc-bcaf-4a67-8edd-beada3347fe5\" (UID: \"b866dadc-bcaf-4a67-8edd-beada3347fe5\") " Oct 31 13:56:08.166209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.166137 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b866dadc-bcaf-4a67-8edd-beada3347fe5-kube-api-access-hs67t\" (UniqueName: \"kubernetes.io/projected/b866dadc-bcaf-4a67-8edd-beada3347fe5-kube-api-access-hs67t\") pod \"b866dadc-bcaf-4a67-8edd-beada3347fe5\" (UID: \"b866dadc-bcaf-4a67-8edd-beada3347fe5\") " Oct 31 13:56:08.166379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.166357 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b866dadc-bcaf-4a67-8edd-beada3347fe5-plugins-config-volume" pod "b866dadc-bcaf-4a67-8edd-beada3347fe5" (UID: "b866dadc-bcaf-4a67-8edd-beada3347fe5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:08.167122 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.167011 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:08.167122 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.167041 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-227pw\" (UniqueName: \"kubernetes.io/projected/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-kube-api-access-227pw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:08.167122 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.167060 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:08.167122 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.167075 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b866dadc-bcaf-4a67-8edd-beada3347fe5-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:08.167122 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.167089 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5e7f54e9-9441-4310-8129-8f7bf99f3bc6-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:08.169102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.169077 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b866dadc-bcaf-4a67-8edd-beada3347fe5-kube-api-access-hs67t" pod "b866dadc-bcaf-4a67-8edd-beada3347fe5" (UID: "b866dadc-bcaf-4a67-8edd-beada3347fe5"). InnerVolumeSpecName "kube-api-access-hs67t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:08.215071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.215041 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" Oct 31 13:56:08.267788 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.267754 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92671137-3b3e-40d4-bb9d-48eabf91adb3-kube-api-access-tkfkf\" (UniqueName: \"kubernetes.io/projected/92671137-3b3e-40d4-bb9d-48eabf91adb3-kube-api-access-tkfkf\") pod \"92671137-3b3e-40d4-bb9d-48eabf91adb3\" (UID: \"92671137-3b3e-40d4-bb9d-48eabf91adb3\") " Oct 31 13:56:08.268236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.268208 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92671137-3b3e-40d4-bb9d-48eabf91adb3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92671137-3b3e-40d4-bb9d-48eabf91adb3-plugins-config-volume\") pod \"92671137-3b3e-40d4-bb9d-48eabf91adb3\" (UID: \"92671137-3b3e-40d4-bb9d-48eabf91adb3\") " Oct 31 13:56:08.268605 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.268569 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92671137-3b3e-40d4-bb9d-48eabf91adb3-plugins-config-volume" pod "92671137-3b3e-40d4-bb9d-48eabf91adb3" (UID: "92671137-3b3e-40d4-bb9d-48eabf91adb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:08.268986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.268967 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92671137-3b3e-40d4-bb9d-48eabf91adb3-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:08.269055 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.268989 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hs67t\" (UniqueName: \"kubernetes.io/projected/b866dadc-bcaf-4a67-8edd-beada3347fe5-kube-api-access-hs67t\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:08.270039 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.270022 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92671137-3b3e-40d4-bb9d-48eabf91adb3-kube-api-access-tkfkf" pod "92671137-3b3e-40d4-bb9d-48eabf91adb3" (UID: "92671137-3b3e-40d4-bb9d-48eabf91adb3"). InnerVolumeSpecName "kube-api-access-tkfkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:08.322964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.322879 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj"] Oct 31 13:56:08.329345 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.329305 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-8a47d6-lhsdj"] Oct 31 13:56:08.371027 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.371001 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkfkf\" (UniqueName: \"kubernetes.io/projected/92671137-3b3e-40d4-bb9d-48eabf91adb3-kube-api-access-tkfkf\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:08.805259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.805224 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" Oct 31 13:56:08.805259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.805243 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj" event={"ID":"92671137-3b3e-40d4-bb9d-48eabf91adb3","Type":"ContainerDied","Data":"73dddb4d91655e01baaba0289c34016bf1903fa80e685ed35cbc86b888deac43"} Oct 31 13:56:08.806288 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.805335 5532 scope.go:122] "RemoveContainer" containerID="22f6bc4e03f2067fcdedbec64254855c0f2b450c166f2e295322ffffd35dbaf3" Oct 31 13:56:08.815613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.807126 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" Oct 31 13:56:08.815613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.807146 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv" event={"ID":"b866dadc-bcaf-4a67-8edd-beada3347fe5","Type":"ContainerDied","Data":"085dd4452797e8163c31f42df2834e360e8cf61ca275d6ac91f4ffda37d7e778"} Oct 31 13:56:08.815613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.809076 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" event={"ID":"5e7f54e9-9441-4310-8129-8f7bf99f3bc6","Type":"ContainerDied","Data":"0d8bfd9f3f929d9a42c08d5f8850e88a14840a68c0ca9a64d26be605b306d5f8"} Oct 31 13:56:08.815613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.809151 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp" Oct 31 13:56:08.820535 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.820514 5532 scope.go:122] "RemoveContainer" containerID="6d5ca2b5d2bbb512e4838aaa2a29ebdd5fccc9de85efbf57aa73f0997acff1dd" Oct 31 13:56:08.832817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.832793 5532 scope.go:122] "RemoveContainer" containerID="5d64728a49adb48d5a5a4cd7037a2f982f826359dec231654ae1bb48edc9949d" Oct 31 13:56:08.889121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.889088 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv"] Oct 31 13:56:08.894311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.894286 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8a47d6-78df5798b8-dxtmv"] Oct 31 13:56:08.904526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.904498 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj"] Oct 31 13:56:08.908927 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.908897 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8a47d6-7556fc4447-wtjfj"] Oct 31 13:56:08.918911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.918888 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp"] Oct 31 13:56:08.923641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:08.923607 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-8a47d6-58857bb5cf-zqjbp"] Oct 31 13:56:09.372698 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:09.372668 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:56:09.385219 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:09.385195 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:56:09.880959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:09.880927 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 13:56:10.006499 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.006462 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0526eecc-8f8c-4ac5-9a54-601cc0e6f692" path="/var/lib/kubelet/pods/0526eecc-8f8c-4ac5-9a54-601cc0e6f692/volumes" Oct 31 13:56:10.007035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.007018 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e7f54e9-9441-4310-8129-8f7bf99f3bc6" path="/var/lib/kubelet/pods/5e7f54e9-9441-4310-8129-8f7bf99f3bc6/volumes" Oct 31 13:56:10.007455 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.007440 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92671137-3b3e-40d4-bb9d-48eabf91adb3" path="/var/lib/kubelet/pods/92671137-3b3e-40d4-bb9d-48eabf91adb3/volumes" Oct 31 13:56:10.008189 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.008172 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b866dadc-bcaf-4a67-8edd-beada3347fe5" path="/var/lib/kubelet/pods/b866dadc-bcaf-4a67-8edd-beada3347fe5/volumes" Oct 31 13:56:10.765896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.765859 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz"] Oct 31 13:56:10.767226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767201 5532 cpu_manager.go:414] "containerMap: removing container" podUID="5e7f54e9-9441-4310-8129-8f7bf99f3bc6" containerName="maas-api" Oct 31 13:56:10.767341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767228 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e7f54e9-9441-4310-8129-8f7bf99f3bc6" containerName="maas-api" Oct 31 13:56:10.767341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767266 5532 cpu_manager.go:414] "containerMap: removing container" podUID="92671137-3b3e-40d4-bb9d-48eabf91adb3" containerName="payload-pre-processing" Oct 31 13:56:10.767341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767274 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92671137-3b3e-40d4-bb9d-48eabf91adb3" containerName="payload-pre-processing" Oct 31 13:56:10.767341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767303 5532 cpu_manager.go:414] "containerMap: removing container" podUID="96216deb-419a-4c46-8132-fb54d26929eb" containerName="payload-pre-processing" Oct 31 13:56:10.767341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767311 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96216deb-419a-4c46-8132-fb54d26929eb" containerName="payload-pre-processing" Oct 31 13:56:10.767341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767323 5532 cpu_manager.go:414] "containerMap: removing container" podUID="9445ef10-8988-4614-80e0-7eb52d5143e1" containerName="payload-processing" Oct 31 13:56:10.767341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767330 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9445ef10-8988-4614-80e0-7eb52d5143e1" containerName="payload-processing" Oct 31 13:56:10.767787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767374 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b866dadc-bcaf-4a67-8edd-beada3347fe5" containerName="payload-processing" Oct 31 13:56:10.767787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767381 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b866dadc-bcaf-4a67-8edd-beada3347fe5" containerName="payload-processing" Oct 31 13:56:10.767787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767619 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e7f54e9-9441-4310-8129-8f7bf99f3bc6" containerName="maas-api" Oct 31 13:56:10.767787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767640 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="92671137-3b3e-40d4-bb9d-48eabf91adb3" containerName="payload-pre-processing" Oct 31 13:56:10.767787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767658 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="96216deb-419a-4c46-8132-fb54d26929eb" containerName="payload-pre-processing" Oct 31 13:56:10.767787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767666 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="9445ef10-8988-4614-80e0-7eb52d5143e1" containerName="payload-processing" Oct 31 13:56:10.767787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.767680 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b866dadc-bcaf-4a67-8edd-beada3347fe5" containerName="payload-processing" Oct 31 13:56:10.774084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.774062 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.783669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.783632 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e50b22-openshift-default-dockercfg-47p7v\"" Oct 31 13:56:10.787910 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.787883 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz"] Oct 31 13:56:10.813733 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.813701 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-data\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.813893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.813796 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-certs\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.813893 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.813870 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w824\" (UniqueName: \"kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-kube-api-access-4w824\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.814029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.813926 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-credential-socket\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.814029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.813948 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c0b065-03aa-4039-beb9-1fcdc146296b-istiod-ca-cert\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.814147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.814057 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-token\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.814147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.814138 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-envoy\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.814268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.814181 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-podinfo\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.814268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.814252 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-socket\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.925616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.919631 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-token\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.925616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.919767 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-envoy\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.925616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.919812 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-podinfo\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.925616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.919912 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-socket\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.925616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.919984 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-data\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.925616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.920078 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-certs\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.925616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.920178 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w824\" (UniqueName: \"kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-kube-api-access-4w824\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.925616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.920244 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-credential-socket\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.925616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.920268 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c0b065-03aa-4039-beb9-1fcdc146296b-istiod-ca-cert\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.936633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.932690 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c0b065-03aa-4039-beb9-1fcdc146296b-istiod-ca-cert\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.936633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.932892 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-socket\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.936633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.932985 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-data\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.936633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.933229 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-certs\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.936633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.934031 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-credential-socket\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.950616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.948995 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-envoy\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.979620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.979562 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-token\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:10.991632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.989510 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w824\" (UniqueName: \"kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-kube-api-access-4w824\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:11.003862 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:10.997207 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-podinfo\") pod \"e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:11.100577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:11.100493 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:11.418126 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:56:11.418093 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c0b065_03aa_4039_beb9_1fcdc146296b.slice/crio-a54cf1de8f932b28e64f1e963c7a1d684e2b885a0cc5d2162bfedd5bee616272 WatchSource:0}: Error finding container a54cf1de8f932b28e64f1e963c7a1d684e2b885a0cc5d2162bfedd5bee616272: Status 404 returned error can't find the container with id a54cf1de8f932b28e64f1e963c7a1d684e2b885a0cc5d2162bfedd5bee616272 Oct 31 13:56:11.419050 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:11.419025 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz"] Oct 31 13:56:11.421154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:11.421124 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:56:11.421223 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:11.421188 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:56:11.421223 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:11.421215 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:56:11.883322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:11.883226 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" event={"ID":"94c0b065-03aa-4039-beb9-1fcdc146296b","Type":"ContainerStarted","Data":"7d48071c015ef2402428926132d720aaa6d1103f71dd7ae22840029cfb80091c"} Oct 31 13:56:11.883322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:11.883291 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" event={"ID":"94c0b065-03aa-4039-beb9-1fcdc146296b","Type":"ContainerStarted","Data":"a54cf1de8f932b28e64f1e963c7a1d684e2b885a0cc5d2162bfedd5bee616272"} Oct 31 13:56:11.914963 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:11.914911 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" podStartSLOduration=1.914896489 podStartE2EDuration="1.914896489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:11.90989421 +0000 UTC m=+3140.055638040" watchObservedRunningTime="2026-10-31 13:56:11.914896489 +0000 UTC m=+3140.060640316" Oct 31 13:56:12.005816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:12.005786 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:12.014004 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:12.013980 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:12.102076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:12.102042 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:12.108857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:12.108829 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:12.892872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:12.892842 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:12.893907 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:12.893878 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:56:12.904188 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:12.904168 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 13:56:15.464243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:15.464199 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29"] Oct 31 13:56:15.469710 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:15.469680 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" Oct 31 13:56:15.476562 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:15.476532 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29"] Oct 31 13:56:15.482856 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:15.482829 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e50b22-dockercfg-9btj4\"" Oct 31 13:56:15.538860 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:15.538835 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e50b22\"" Oct 31 13:56:15.612616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:15.611870 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj7h6\" (UniqueName: \"kubernetes.io/projected/82327512-b6dc-45ab-8d1c-3453cfa8bd15-kube-api-access-lj7h6\") pod \"payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29\" (UID: \"82327512-b6dc-45ab-8d1c-3453cfa8bd15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" Oct 31 13:56:15.612616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:15.612094 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82327512-b6dc-45ab-8d1c-3453cfa8bd15-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29\" (UID: \"82327512-b6dc-45ab-8d1c-3453cfa8bd15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" Oct 31 13:56:15.718121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:15.716877 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82327512-b6dc-45ab-8d1c-3453cfa8bd15-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29\" (UID: \"82327512-b6dc-45ab-8d1c-3453cfa8bd15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" Oct 31 13:56:15.718479 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:15.718032 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82327512-b6dc-45ab-8d1c-3453cfa8bd15-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29\" (UID: \"82327512-b6dc-45ab-8d1c-3453cfa8bd15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" Oct 31 13:56:15.718897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:15.718875 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj7h6\" (UniqueName: \"kubernetes.io/projected/82327512-b6dc-45ab-8d1c-3453cfa8bd15-kube-api-access-lj7h6\") pod \"payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29\" (UID: \"82327512-b6dc-45ab-8d1c-3453cfa8bd15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" Oct 31 13:56:15.737821 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:15.737757 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj7h6\" (UniqueName: \"kubernetes.io/projected/82327512-b6dc-45ab-8d1c-3453cfa8bd15-kube-api-access-lj7h6\") pod \"payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29\" (UID: \"82327512-b6dc-45ab-8d1c-3453cfa8bd15\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" Oct 31 13:56:15.792577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:15.792546 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" Oct 31 13:56:16.089802 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:56:16.089762 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82327512_b6dc_45ab_8d1c_3453cfa8bd15.slice/crio-1f4a097575384b5cec5af8a119e100cf10346217e9dd3f8dea6320ef861a22ad WatchSource:0}: Error finding container 1f4a097575384b5cec5af8a119e100cf10346217e9dd3f8dea6320ef861a22ad: Status 404 returned error can't find the container with id 1f4a097575384b5cec5af8a119e100cf10346217e9dd3f8dea6320ef861a22ad Oct 31 13:56:16.091750 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:16.091721 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29"] Oct 31 13:56:16.596657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:16.596618 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn"] Oct 31 13:56:16.597162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:16.596881 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" containerID="cri-o://350796dabd8abc7f9e629f8689575fbe71290caf9f8d923b362475dfd3f8f107" gracePeriod=30 Oct 31 13:56:16.941651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:16.941311 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" event={"ID":"82327512-b6dc-45ab-8d1c-3453cfa8bd15","Type":"ContainerStarted","Data":"ad85c6467af98caf4948f8f1ce7f7bb488a50e8b95df12057a89eebc9823a8fc"} Oct 31 13:56:16.941651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:16.941372 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" event={"ID":"82327512-b6dc-45ab-8d1c-3453cfa8bd15","Type":"ContainerStarted","Data":"1f4a097575384b5cec5af8a119e100cf10346217e9dd3f8dea6320ef861a22ad"} Oct 31 13:56:16.941651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:16.941405 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" Oct 31 13:56:16.964645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:16.964570 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" podStartSLOduration=1.9645542329999999 podStartE2EDuration="1.964554233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:16.962536331 +0000 UTC m=+3145.108280158" watchObservedRunningTime="2026-10-31 13:56:16.964554233 +0000 UTC m=+3145.110298064" Oct 31 13:56:17.031586 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.031550 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw"] Oct 31 13:56:17.041505 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.041478 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8622936-7b7kw"] Oct 31 13:56:17.276049 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.275970 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn"] Oct 31 13:56:17.280878 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.280858 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" Oct 31 13:56:17.285747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.285726 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:56:17.285953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.285933 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:56:17.290389 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.290364 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn"] Oct 31 13:56:17.389889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.389841 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppz88\" (UniqueName: \"kubernetes.io/projected/9457b600-c4fd-4784-ac44-c348292258c4-kube-api-access-ppz88\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn\" (UID: \"9457b600-c4fd-4784-ac44-c348292258c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" Oct 31 13:56:17.390102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.389968 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn\" (UID: \"9457b600-c4fd-4784-ac44-c348292258c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" Oct 31 13:56:17.390102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.390059 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn\" (UID: \"9457b600-c4fd-4784-ac44-c348292258c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" Oct 31 13:56:17.495057 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.495018 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn\" (UID: \"9457b600-c4fd-4784-ac44-c348292258c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" Oct 31 13:56:17.496075 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.496041 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn\" (UID: \"9457b600-c4fd-4784-ac44-c348292258c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" Oct 31 13:56:17.499187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.499152 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn\" (UID: \"9457b600-c4fd-4784-ac44-c348292258c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" Oct 31 13:56:17.499187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.499116 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn\" (UID: \"9457b600-c4fd-4784-ac44-c348292258c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" Oct 31 13:56:17.499569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.499545 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppz88\" (UniqueName: \"kubernetes.io/projected/9457b600-c4fd-4784-ac44-c348292258c4-kube-api-access-ppz88\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn\" (UID: \"9457b600-c4fd-4784-ac44-c348292258c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" Oct 31 13:56:17.514100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.514072 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppz88\" (UniqueName: \"kubernetes.io/projected/9457b600-c4fd-4784-ac44-c348292258c4-kube-api-access-ppz88\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn\" (UID: \"9457b600-c4fd-4784-ac44-c348292258c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" Oct 31 13:56:17.598793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.598707 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" Oct 31 13:56:17.889020 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:56:17.888985 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9457b600_c4fd_4784_ac44_c348292258c4.slice/crio-528db761743f15a6f0066eca58788d7710c0fb666a0d71fc261f32e60e05b75f WatchSource:0}: Error finding container 528db761743f15a6f0066eca58788d7710c0fb666a0d71fc261f32e60e05b75f: Status 404 returned error can't find the container with id 528db761743f15a6f0066eca58788d7710c0fb666a0d71fc261f32e60e05b75f Oct 31 13:56:17.890467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.890436 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn"] Oct 31 13:56:17.951781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:17.951752 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" event={"ID":"9457b600-c4fd-4784-ac44-c348292258c4","Type":"ContainerStarted","Data":"528db761743f15a6f0066eca58788d7710c0fb666a0d71fc261f32e60e05b75f"} Oct 31 13:56:18.007492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:18.007461 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6" path="/var/lib/kubelet/pods/eb726aed-e9aa-4bb2-98ac-3f3ddc8686a6/volumes" Oct 31 13:56:18.961390 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:18.961354 5532 generic.go:356] "Generic (PLEG): container finished" podID="9457b600-c4fd-4784-ac44-c348292258c4" containerID="e71eb27584eae16ed7146bc8886e912adc129c73d0884cf7a9a8486763b4314d" exitCode=0 Oct 31 13:56:18.961802 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:18.961444 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" event={"ID":"9457b600-c4fd-4784-ac44-c348292258c4","Type":"ContainerDied","Data":"e71eb27584eae16ed7146bc8886e912adc129c73d0884cf7a9a8486763b4314d"} Oct 31 13:56:20.138789 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.138760 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz"] Oct 31 13:56:20.145170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.145147 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" Oct 31 13:56:20.149981 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.149956 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-dafbce-dockercfg-4lqff\"" Oct 31 13:56:20.150127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.149956 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-dafbce\"" Oct 31 13:56:20.151028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.151005 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz"] Oct 31 13:56:20.157699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.157679 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" Oct 31 13:56:20.423216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.409778 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw"] Oct 31 13:56:20.423216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.411099 5532 cpu_manager.go:414] "containerMap: removing container" podUID="9457b600-c4fd-4784-ac44-c348292258c4" containerName="revoke-keys" Oct 31 13:56:20.423216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.411122 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9457b600-c4fd-4784-ac44-c348292258c4" containerName="revoke-keys" Oct 31 13:56:20.423216 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.411309 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="9457b600-c4fd-4784-ac44-c348292258c4" containerName="revoke-keys" Oct 31 13:56:20.431651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.431570 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw"] Oct 31 13:56:20.431930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.431725 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" Oct 31 13:56:20.505052 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.505008 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-xks-ca\") pod \"9457b600-c4fd-4784-ac44-c348292258c4\" (UID: \"9457b600-c4fd-4784-ac44-c348292258c4\") " Oct 31 13:56:20.505242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.505062 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9457b600-c4fd-4784-ac44-c348292258c4-kube-api-access-ppz88\" (UniqueName: \"kubernetes.io/projected/9457b600-c4fd-4784-ac44-c348292258c4-kube-api-access-ppz88\") pod \"9457b600-c4fd-4784-ac44-c348292258c4\" (UID: \"9457b600-c4fd-4784-ac44-c348292258c4\") " Oct 31 13:56:20.505354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.505332 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-openshift-ca\") pod \"9457b600-c4fd-4784-ac44-c348292258c4\" (UID: \"9457b600-c4fd-4784-ac44-c348292258c4\") " Oct 31 13:56:20.505761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.505729 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-openshift-ca" pod "9457b600-c4fd-4784-ac44-c348292258c4" (UID: "9457b600-c4fd-4784-ac44-c348292258c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:20.505942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.505918 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b2679c-6948-4251-aec3-cdd30eb68497-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw\" (UID: \"b4b2679c-6948-4251-aec3-cdd30eb68497\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" Oct 31 13:56:20.506025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.505982 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55jm2\" (UniqueName: \"kubernetes.io/projected/f6a13f9a-dafa-4839-ac03-247b821dd5c7-kube-api-access-55jm2\") pod \"payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz\" (UID: \"f6a13f9a-dafa-4839-ac03-247b821dd5c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" Oct 31 13:56:20.506192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.506170 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a13f9a-dafa-4839-ac03-247b821dd5c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz\" (UID: \"f6a13f9a-dafa-4839-ac03-247b821dd5c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" Oct 31 13:56:20.506294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.506270 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqr9h\" (UniqueName: \"kubernetes.io/projected/b4b2679c-6948-4251-aec3-cdd30eb68497-kube-api-access-wqr9h\") pod \"payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw\" (UID: \"b4b2679c-6948-4251-aec3-cdd30eb68497\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" Oct 31 13:56:20.506461 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.506448 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:20.507353 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.507325 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9457b600-c4fd-4784-ac44-c348292258c4-kube-api-access-ppz88" pod "9457b600-c4fd-4784-ac44-c348292258c4" (UID: "9457b600-c4fd-4784-ac44-c348292258c4"). InnerVolumeSpecName "kube-api-access-ppz88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:20.507466 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.507389 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-xks-ca" pod "9457b600-c4fd-4784-ac44-c348292258c4" (UID: "9457b600-c4fd-4784-ac44-c348292258c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:56:20.607431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.607389 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55jm2\" (UniqueName: \"kubernetes.io/projected/f6a13f9a-dafa-4839-ac03-247b821dd5c7-kube-api-access-55jm2\") pod \"payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz\" (UID: \"f6a13f9a-dafa-4839-ac03-247b821dd5c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" Oct 31 13:56:20.607678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.607578 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a13f9a-dafa-4839-ac03-247b821dd5c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz\" (UID: \"f6a13f9a-dafa-4839-ac03-247b821dd5c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" Oct 31 13:56:20.607755 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.607724 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqr9h\" (UniqueName: \"kubernetes.io/projected/b4b2679c-6948-4251-aec3-cdd30eb68497-kube-api-access-wqr9h\") pod \"payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw\" (UID: \"b4b2679c-6948-4251-aec3-cdd30eb68497\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" Oct 31 13:56:20.607937 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.607919 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b2679c-6948-4251-aec3-cdd30eb68497-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw\" (UID: \"b4b2679c-6948-4251-aec3-cdd30eb68497\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" Oct 31 13:56:20.608088 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.608073 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9457b600-c4fd-4784-ac44-c348292258c4-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:20.608148 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.608094 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppz88\" (UniqueName: \"kubernetes.io/projected/9457b600-c4fd-4784-ac44-c348292258c4-kube-api-access-ppz88\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:20.608478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.608450 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a13f9a-dafa-4839-ac03-247b821dd5c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz\" (UID: \"f6a13f9a-dafa-4839-ac03-247b821dd5c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" Oct 31 13:56:20.608739 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.608719 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b2679c-6948-4251-aec3-cdd30eb68497-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw\" (UID: \"b4b2679c-6948-4251-aec3-cdd30eb68497\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" Oct 31 13:56:20.623019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.622989 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqr9h\" (UniqueName: \"kubernetes.io/projected/b4b2679c-6948-4251-aec3-cdd30eb68497-kube-api-access-wqr9h\") pod \"payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw\" (UID: \"b4b2679c-6948-4251-aec3-cdd30eb68497\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" Oct 31 13:56:20.623163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.623056 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55jm2\" (UniqueName: \"kubernetes.io/projected/f6a13f9a-dafa-4839-ac03-247b821dd5c7-kube-api-access-55jm2\") pod \"payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz\" (UID: \"f6a13f9a-dafa-4839-ac03-247b821dd5c7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" Oct 31 13:56:20.748409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.748336 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" Oct 31 13:56:20.772128 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.772104 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" Oct 31 13:56:20.999967 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.999871 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" event={"ID":"9457b600-c4fd-4784-ac44-c348292258c4","Type":"ContainerDied","Data":"528db761743f15a6f0066eca58788d7710c0fb666a0d71fc261f32e60e05b75f"} Oct 31 13:56:20.999967 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:20.999942 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="528db761743f15a6f0066eca58788d7710c0fb666a0d71fc261f32e60e05b75f" Oct 31 13:56:21.000210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:21.000018 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn" Oct 31 13:56:21.041737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:21.041710 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw"] Oct 31 13:56:21.041990 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:56:21.041969 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4b2679c_6948_4251_aec3_cdd30eb68497.slice/crio-007fe33f7911dd3eaf23119a3649e741971208d02a45f8ff526d89bb1a5bb863 WatchSource:0}: Error finding container 007fe33f7911dd3eaf23119a3649e741971208d02a45f8ff526d89bb1a5bb863: Status 404 returned error can't find the container with id 007fe33f7911dd3eaf23119a3649e741971208d02a45f8ff526d89bb1a5bb863 Oct 31 13:56:21.079871 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:21.079845 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz"] Oct 31 13:56:21.932100 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:21.932077 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:56:22.018569 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.018540 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" Oct 31 13:56:22.018803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.018609 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" event={"ID":"f6a13f9a-dafa-4839-ac03-247b821dd5c7","Type":"ContainerStarted","Data":"323b94ff4cfd11c0d526402f90e94431fe6ba873b89612c5aea0867fe2333482"} Oct 31 13:56:22.018803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.018637 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" event={"ID":"f6a13f9a-dafa-4839-ac03-247b821dd5c7","Type":"ContainerStarted","Data":"de7b8ab404f6ce700ca8828717ca79777851ed3423bae6048a66986c3e7d0f1e"} Oct 31 13:56:22.020093 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.020061 5532 generic.go:356] "Generic (PLEG): container finished" podID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerID="350796dabd8abc7f9e629f8689575fbe71290caf9f8d923b362475dfd3f8f107" exitCode=0 Oct 31 13:56:22.020251 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.020137 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" Oct 31 13:56:22.020251 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.020151 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" event={"ID":"dccbf2fa-9e8d-453b-ba9d-60dae20a4884","Type":"ContainerDied","Data":"350796dabd8abc7f9e629f8689575fbe71290caf9f8d923b362475dfd3f8f107"} Oct 31 13:56:22.020251 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.020200 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn" event={"ID":"dccbf2fa-9e8d-453b-ba9d-60dae20a4884","Type":"ContainerDied","Data":"4274105f86bdf17174661ba6d34900a9cb9790910927e1840e0ba671e238b876"} Oct 31 13:56:22.020251 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.020232 5532 scope.go:122] "RemoveContainer" containerID="350796dabd8abc7f9e629f8689575fbe71290caf9f8d923b362475dfd3f8f107" Oct 31 13:56:22.022794 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.022766 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" event={"ID":"b4b2679c-6948-4251-aec3-cdd30eb68497","Type":"ContainerStarted","Data":"b951b0c7bc811951f85162a57d431d6350dde05372b03d0f332ce7fab26698f0"} Oct 31 13:56:22.022888 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.022817 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" event={"ID":"b4b2679c-6948-4251-aec3-cdd30eb68497","Type":"ContainerStarted","Data":"007fe33f7911dd3eaf23119a3649e741971208d02a45f8ff526d89bb1a5bb863"} Oct 31 13:56:22.022940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.022888 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" Oct 31 13:56:22.034040 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.034019 5532 scope.go:122] "RemoveContainer" containerID="92ffbb7a5f36d38e286f158c59ff493e16dc49698bfa76fe7ad991a1a7167a65" Oct 31 13:56:22.034315 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.034285 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-certs\") pod \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " Oct 31 13:56:22.034402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.034326 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-credential-socket\") pod \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " Oct 31 13:56:22.034402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.034374 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-data\") pod \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " Oct 31 13:56:22.034532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.034444 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-token\" (UniqueName: \"kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-token\") pod \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " Oct 31 13:56:22.034532 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.034505 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-podinfo\") pod \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " Oct 31 13:56:22.034660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.034541 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istiod-ca-cert\") pod \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " Oct 31 13:56:22.034742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.034719 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-data" pod "dccbf2fa-9e8d-453b-ba9d-60dae20a4884" (UID: "dccbf2fa-9e8d-453b-ba9d-60dae20a4884"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:56:22.035024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.034585 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-envoy\") pod \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " Oct 31 13:56:22.035024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.035011 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istiod-ca-cert" pod "dccbf2fa-9e8d-453b-ba9d-60dae20a4884" (UID: "dccbf2fa-9e8d-453b-ba9d-60dae20a4884"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:22.035189 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.035125 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-kube-api-access-rgckm\" (UniqueName: \"kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-kube-api-access-rgckm\") pod \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " Oct 31 13:56:22.035253 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.035126 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-credential-socket" pod "dccbf2fa-9e8d-453b-ba9d-60dae20a4884" (UID: "dccbf2fa-9e8d-453b-ba9d-60dae20a4884"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:56:22.035307 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.035295 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-socket\") pod \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\" (UID: \"dccbf2fa-9e8d-453b-ba9d-60dae20a4884\") " Oct 31 13:56:22.035411 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.035389 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-certs" pod "dccbf2fa-9e8d-453b-ba9d-60dae20a4884" (UID: "dccbf2fa-9e8d-453b-ba9d-60dae20a4884"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:56:22.035838 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.035790 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-socket" pod "dccbf2fa-9e8d-453b-ba9d-60dae20a4884" (UID: "dccbf2fa-9e8d-453b-ba9d-60dae20a4884"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:56:22.038358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.038337 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:22.038358 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.038358 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:22.038527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.038368 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:22.038527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.038378 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:22.038527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.038387 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:22.040700 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.040661 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-token" pod "dccbf2fa-9e8d-453b-ba9d-60dae20a4884" (UID: "dccbf2fa-9e8d-453b-ba9d-60dae20a4884"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:22.041117 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.041093 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-envoy" pod "dccbf2fa-9e8d-453b-ba9d-60dae20a4884" (UID: "dccbf2fa-9e8d-453b-ba9d-60dae20a4884"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:56:22.041438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.041401 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-podinfo" pod "dccbf2fa-9e8d-453b-ba9d-60dae20a4884" (UID: "dccbf2fa-9e8d-453b-ba9d-60dae20a4884"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:56:22.043993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.043930 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" podStartSLOduration=2.043914303 podStartE2EDuration="2.043914303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:22.0375674 +0000 UTC m=+3150.183311231" watchObservedRunningTime="2026-10-31 13:56:22.043914303 +0000 UTC m=+3150.189658135" Oct 31 13:56:22.048288 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.048263 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-kube-api-access-rgckm" pod "dccbf2fa-9e8d-453b-ba9d-60dae20a4884" (UID: "dccbf2fa-9e8d-453b-ba9d-60dae20a4884"). InnerVolumeSpecName "kube-api-access-rgckm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:22.063139 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.061996 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" podStartSLOduration=2.061977978 podStartE2EDuration="2.061977978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:22.059006713 +0000 UTC m=+3150.204750544" watchObservedRunningTime="2026-10-31 13:56:22.061977978 +0000 UTC m=+3150.207721810" Oct 31 13:56:22.094092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.088310 5532 scope.go:122] "RemoveContainer" containerID="350796dabd8abc7f9e629f8689575fbe71290caf9f8d923b362475dfd3f8f107" Oct 31 13:56:22.094092 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:56:22.088927 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"350796dabd8abc7f9e629f8689575fbe71290caf9f8d923b362475dfd3f8f107\": container with ID starting with 350796dabd8abc7f9e629f8689575fbe71290caf9f8d923b362475dfd3f8f107 not found: ID does not exist" containerID="350796dabd8abc7f9e629f8689575fbe71290caf9f8d923b362475dfd3f8f107" Oct 31 13:56:22.094092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.088963 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"350796dabd8abc7f9e629f8689575fbe71290caf9f8d923b362475dfd3f8f107"} err="failed to get container status \"350796dabd8abc7f9e629f8689575fbe71290caf9f8d923b362475dfd3f8f107\": rpc error: code = NotFound desc = could not find container \"350796dabd8abc7f9e629f8689575fbe71290caf9f8d923b362475dfd3f8f107\": container with ID starting with 350796dabd8abc7f9e629f8689575fbe71290caf9f8d923b362475dfd3f8f107 not found: ID does not exist" Oct 31 13:56:22.094092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.088986 5532 scope.go:122] "RemoveContainer" containerID="92ffbb7a5f36d38e286f158c59ff493e16dc49698bfa76fe7ad991a1a7167a65" Oct 31 13:56:22.094092 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:56:22.089276 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92ffbb7a5f36d38e286f158c59ff493e16dc49698bfa76fe7ad991a1a7167a65\": container with ID starting with 92ffbb7a5f36d38e286f158c59ff493e16dc49698bfa76fe7ad991a1a7167a65 not found: ID does not exist" containerID="92ffbb7a5f36d38e286f158c59ff493e16dc49698bfa76fe7ad991a1a7167a65" Oct 31 13:56:22.094092 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.089314 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92ffbb7a5f36d38e286f158c59ff493e16dc49698bfa76fe7ad991a1a7167a65"} err="failed to get container status \"92ffbb7a5f36d38e286f158c59ff493e16dc49698bfa76fe7ad991a1a7167a65\": rpc error: code = NotFound desc = could not find container \"92ffbb7a5f36d38e286f158c59ff493e16dc49698bfa76fe7ad991a1a7167a65\": container with ID starting with 92ffbb7a5f36d38e286f158c59ff493e16dc49698bfa76fe7ad991a1a7167a65 not found: ID does not exist" Oct 31 13:56:22.139701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.139671 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:22.139701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.139702 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:22.139961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.139718 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:22.139961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.139732 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgckm\" (UniqueName: \"kubernetes.io/projected/dccbf2fa-9e8d-453b-ba9d-60dae20a4884-kube-api-access-rgckm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:22.360629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.360587 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn"] Oct 31 13:56:22.367687 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:22.367657 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8a47d6-openshift-default-5b466b7b45-qgqcn"] Oct 31 13:56:24.015894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.015858 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" path="/var/lib/kubelet/pods/dccbf2fa-9e8d-453b-ba9d-60dae20a4884/volumes" Oct 31 13:56:24.016307 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.016285 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567"] Oct 31 13:56:24.017124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.017107 5532 cpu_manager.go:414] "containerMap: removing container" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" Oct 31 13:56:24.017181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.017126 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" Oct 31 13:56:24.017181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.017139 5532 cpu_manager.go:414] "containerMap: removing container" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" Oct 31 13:56:24.017181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.017144 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" Oct 31 13:56:24.017279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.017254 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" Oct 31 13:56:24.017279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.017263 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccbf2fa-9e8d-453b-ba9d-60dae20a4884" containerName="istio-proxy" Oct 31 13:56:24.024136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.024111 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.026727 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.026699 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567"] Oct 31 13:56:24.036962 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.036919 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 31 13:56:24.037106 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.037006 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 31 13:56:24.037106 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.037070 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 31 13:56:24.037106 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.036946 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zfm9b\"" Oct 31 13:56:24.071803 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.071765 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.071989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.071886 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9gg\" (UniqueName: \"kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-kube-api-access-lm9gg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.072062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.072033 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.072518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.072489 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.072738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.072560 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e425c2-8c42-49f3-ba02-9e36514b53eb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.072738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.072637 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.073051 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.073028 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.174696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.174650 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9gg\" (UniqueName: \"kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-kube-api-access-lm9gg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.174915 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.174811 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.174991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.174933 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.174991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.174969 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e425c2-8c42-49f3-ba02-9e36514b53eb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.175096 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.175014 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.175153 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.175140 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.175233 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.175212 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.175840 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.175802 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.178547 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.178426 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.181514 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.181487 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.181992 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.181938 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.183212 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.183174 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e425c2-8c42-49f3-ba02-9e36514b53eb-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.197489 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.197458 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9gg\" (UniqueName: \"kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-kube-api-access-lm9gg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.198252 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.198224 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-gk567\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.347656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.347548 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:24.646989 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:56:24.646956 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e425c2_8c42_49f3_ba02_9e36514b53eb.slice/crio-ed21b3e452a4e4d46f061aa91b49db656dd0efbd5ccfa2c94844328403e7d806 WatchSource:0}: Error finding container ed21b3e452a4e4d46f061aa91b49db656dd0efbd5ccfa2c94844328403e7d806: Status 404 returned error can't find the container with id ed21b3e452a4e4d46f061aa91b49db656dd0efbd5ccfa2c94844328403e7d806 Oct 31 13:56:24.648436 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.648408 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567"] Oct 31 13:56:24.650442 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.650405 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:56:24.650543 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:24.650485 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:56:25.079027 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:25.078990 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" event={"ID":"18e425c2-8c42-49f3-ba02-9e36514b53eb","Type":"ContainerStarted","Data":"d3acd17ff8a44cdbadc70a106eb407e15f89b96ffd5db774a9758cf336002f7d"} Oct 31 13:56:25.079557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:25.079051 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" event={"ID":"18e425c2-8c42-49f3-ba02-9e36514b53eb","Type":"ContainerStarted","Data":"ed21b3e452a4e4d46f061aa91b49db656dd0efbd5ccfa2c94844328403e7d806"} Oct 31 13:56:25.079557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:25.079072 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:25.081987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:25.081954 5532 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-gk567 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:56:25.082124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:25.082012 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" podUID="18e425c2-8c42-49f3-ba02-9e36514b53eb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:56:25.114268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:25.114210 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" podStartSLOduration=2.114193972 podStartE2EDuration="2.114193972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:56:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:56:25.104198095 +0000 UTC m=+3153.249941952" watchObservedRunningTime="2026-10-31 13:56:25.114193972 +0000 UTC m=+3153.259937799" Oct 31 13:56:26.089547 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:26.089515 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 13:56:27.481239 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:27.481203 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d"] Oct 31 13:56:27.494837 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:27.494807 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4"] Oct 31 13:56:27.956720 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:27.956688 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" Oct 31 13:56:31.856330 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:56:31.856252 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:56:32.481760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.481714 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" podUID="01c8f8ae-be64-4501-adb5-dc2689f938bf" containerName="payload-processing" containerID="cri-o://7b31504b1944e6e8a0f48b52c5a94a88a733e7f43ce514929fc6bd5d00dc9f63" gracePeriod=25 Oct 31 13:56:32.495575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.495517 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" podUID="2cce49b5-22bd-4655-90eb-75b5fb6cae5b" containerName="payload-pre-processing" containerID="cri-o://c3a59b3d1bf64ecda52acdb1eb1280f9995472a7f35d5f731b3461a09df3fcc0" gracePeriod=25 Oct 31 13:56:32.797533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.797502 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" Oct 31 13:56:32.863441 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.863380 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" Oct 31 13:56:32.952757 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.952728 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01c8f8ae-be64-4501-adb5-dc2689f938bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c8f8ae-be64-4501-adb5-dc2689f938bf-plugins-config-volume\") pod \"01c8f8ae-be64-4501-adb5-dc2689f938bf\" (UID: \"01c8f8ae-be64-4501-adb5-dc2689f938bf\") " Oct 31 13:56:32.952948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.952771 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-plugins-config-volume\") pod \"2cce49b5-22bd-4655-90eb-75b5fb6cae5b\" (UID: \"2cce49b5-22bd-4655-90eb-75b5fb6cae5b\") " Oct 31 13:56:32.952948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.952833 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-kube-api-access-qfbk2\" (UniqueName: \"kubernetes.io/projected/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-kube-api-access-qfbk2\") pod \"2cce49b5-22bd-4655-90eb-75b5fb6cae5b\" (UID: \"2cce49b5-22bd-4655-90eb-75b5fb6cae5b\") " Oct 31 13:56:32.952948 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.952859 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01c8f8ae-be64-4501-adb5-dc2689f938bf-kube-api-access-pz9nl\" (UniqueName: \"kubernetes.io/projected/01c8f8ae-be64-4501-adb5-dc2689f938bf-kube-api-access-pz9nl\") pod \"01c8f8ae-be64-4501-adb5-dc2689f938bf\" (UID: \"01c8f8ae-be64-4501-adb5-dc2689f938bf\") " Oct 31 13:56:32.953256 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.953104 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01c8f8ae-be64-4501-adb5-dc2689f938bf-plugins-config-volume" pod "01c8f8ae-be64-4501-adb5-dc2689f938bf" (UID: "01c8f8ae-be64-4501-adb5-dc2689f938bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:32.953324 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.953274 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-plugins-config-volume" pod "2cce49b5-22bd-4655-90eb-75b5fb6cae5b" (UID: "2cce49b5-22bd-4655-90eb-75b5fb6cae5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:32.953681 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.953661 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01c8f8ae-be64-4501-adb5-dc2689f938bf-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:32.953736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.953684 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:32.955030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.955012 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-kube-api-access-qfbk2" pod "2cce49b5-22bd-4655-90eb-75b5fb6cae5b" (UID: "2cce49b5-22bd-4655-90eb-75b5fb6cae5b"). InnerVolumeSpecName "kube-api-access-qfbk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:32.955071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:32.955042 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01c8f8ae-be64-4501-adb5-dc2689f938bf-kube-api-access-pz9nl" pod "01c8f8ae-be64-4501-adb5-dc2689f938bf" (UID: "01c8f8ae-be64-4501-adb5-dc2689f938bf"). InnerVolumeSpecName "kube-api-access-pz9nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:33.038769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.038675 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" Oct 31 13:56:33.038769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.038757 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" Oct 31 13:56:33.055104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.055072 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfbk2\" (UniqueName: \"kubernetes.io/projected/2cce49b5-22bd-4655-90eb-75b5fb6cae5b-kube-api-access-qfbk2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:33.055104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.055100 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz9nl\" (UniqueName: \"kubernetes.io/projected/01c8f8ae-be64-4501-adb5-dc2689f938bf-kube-api-access-pz9nl\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:33.185116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.185077 5532 generic.go:356] "Generic (PLEG): container finished" podID="01c8f8ae-be64-4501-adb5-dc2689f938bf" containerID="7b31504b1944e6e8a0f48b52c5a94a88a733e7f43ce514929fc6bd5d00dc9f63" exitCode=0 Oct 31 13:56:33.185116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.185125 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" Oct 31 13:56:33.185350 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.185159 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" event={"ID":"01c8f8ae-be64-4501-adb5-dc2689f938bf","Type":"ContainerDied","Data":"7b31504b1944e6e8a0f48b52c5a94a88a733e7f43ce514929fc6bd5d00dc9f63"} Oct 31 13:56:33.185350 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.185214 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d" event={"ID":"01c8f8ae-be64-4501-adb5-dc2689f938bf","Type":"ContainerDied","Data":"f517e42202a9501751db3d304f11d508a92530aab071a7be676d475e38bfc541"} Oct 31 13:56:33.185350 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.185235 5532 scope.go:122] "RemoveContainer" containerID="7b31504b1944e6e8a0f48b52c5a94a88a733e7f43ce514929fc6bd5d00dc9f63" Oct 31 13:56:33.186936 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.186910 5532 generic.go:356] "Generic (PLEG): container finished" podID="2cce49b5-22bd-4655-90eb-75b5fb6cae5b" containerID="c3a59b3d1bf64ecda52acdb1eb1280f9995472a7f35d5f731b3461a09df3fcc0" exitCode=0 Oct 31 13:56:33.187079 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.186961 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" Oct 31 13:56:33.187079 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.186990 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" event={"ID":"2cce49b5-22bd-4655-90eb-75b5fb6cae5b","Type":"ContainerDied","Data":"c3a59b3d1bf64ecda52acdb1eb1280f9995472a7f35d5f731b3461a09df3fcc0"} Oct 31 13:56:33.187079 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.187037 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4" event={"ID":"2cce49b5-22bd-4655-90eb-75b5fb6cae5b","Type":"ContainerDied","Data":"85a029ec973da22f667477e40d794e3e58a545b2491cda14fbfbd929bfc90a4c"} Oct 31 13:56:33.206704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.206685 5532 scope.go:122] "RemoveContainer" containerID="7b31504b1944e6e8a0f48b52c5a94a88a733e7f43ce514929fc6bd5d00dc9f63" Oct 31 13:56:33.207023 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:56:33.207001 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b31504b1944e6e8a0f48b52c5a94a88a733e7f43ce514929fc6bd5d00dc9f63\": container with ID starting with 7b31504b1944e6e8a0f48b52c5a94a88a733e7f43ce514929fc6bd5d00dc9f63 not found: ID does not exist" containerID="7b31504b1944e6e8a0f48b52c5a94a88a733e7f43ce514929fc6bd5d00dc9f63" Oct 31 13:56:33.207098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.207032 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b31504b1944e6e8a0f48b52c5a94a88a733e7f43ce514929fc6bd5d00dc9f63"} err="failed to get container status \"7b31504b1944e6e8a0f48b52c5a94a88a733e7f43ce514929fc6bd5d00dc9f63\": rpc error: code = NotFound desc = could not find container \"7b31504b1944e6e8a0f48b52c5a94a88a733e7f43ce514929fc6bd5d00dc9f63\": container with ID starting with 7b31504b1944e6e8a0f48b52c5a94a88a733e7f43ce514929fc6bd5d00dc9f63 not found: ID does not exist" Oct 31 13:56:33.207098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.207049 5532 scope.go:122] "RemoveContainer" containerID="c3a59b3d1bf64ecda52acdb1eb1280f9995472a7f35d5f731b3461a09df3fcc0" Oct 31 13:56:33.219237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.219139 5532 scope.go:122] "RemoveContainer" containerID="c3a59b3d1bf64ecda52acdb1eb1280f9995472a7f35d5f731b3461a09df3fcc0" Oct 31 13:56:33.219450 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:56:33.219428 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3a59b3d1bf64ecda52acdb1eb1280f9995472a7f35d5f731b3461a09df3fcc0\": container with ID starting with c3a59b3d1bf64ecda52acdb1eb1280f9995472a7f35d5f731b3461a09df3fcc0 not found: ID does not exist" containerID="c3a59b3d1bf64ecda52acdb1eb1280f9995472a7f35d5f731b3461a09df3fcc0" Oct 31 13:56:33.219526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.219458 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3a59b3d1bf64ecda52acdb1eb1280f9995472a7f35d5f731b3461a09df3fcc0"} err="failed to get container status \"c3a59b3d1bf64ecda52acdb1eb1280f9995472a7f35d5f731b3461a09df3fcc0\": rpc error: code = NotFound desc = could not find container \"c3a59b3d1bf64ecda52acdb1eb1280f9995472a7f35d5f731b3461a09df3fcc0\": container with ID starting with c3a59b3d1bf64ecda52acdb1eb1280f9995472a7f35d5f731b3461a09df3fcc0 not found: ID does not exist" Oct 31 13:56:33.224300 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.224272 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4"] Oct 31 13:56:33.236106 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.236085 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-4f340c-5b7cf9b475-9ghl4"] Oct 31 13:56:33.248709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.248684 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d"] Oct 31 13:56:33.256377 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.256341 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-4f340c-74b9cf49cc-56k6d"] Oct 31 13:56:33.930525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.930490 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl"] Oct 31 13:56:33.931084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:33.930794 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" podUID="aed42dc6-80f5-4a58-9b3d-ffca597ddef5" containerName="maas-api" containerID="cri-o://e870d6c9245fb909fd6eac252a527ffe67173c30a94fcae75b22ce712becff5d" gracePeriod=30 Oct 31 13:56:34.006286 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.006239 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01c8f8ae-be64-4501-adb5-dc2689f938bf" path="/var/lib/kubelet/pods/01c8f8ae-be64-4501-adb5-dc2689f938bf/volumes" Oct 31 13:56:34.006835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.006812 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cce49b5-22bd-4655-90eb-75b5fb6cae5b" path="/var/lib/kubelet/pods/2cce49b5-22bd-4655-90eb-75b5fb6cae5b/volumes" Oct 31 13:56:34.365721 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.365680 5532 generic.go:356] "Generic (PLEG): container finished" podID="aed42dc6-80f5-4a58-9b3d-ffca597ddef5" containerID="e870d6c9245fb909fd6eac252a527ffe67173c30a94fcae75b22ce712becff5d" exitCode=0 Oct 31 13:56:34.366064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.366041 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" event={"ID":"aed42dc6-80f5-4a58-9b3d-ffca597ddef5","Type":"ContainerDied","Data":"e870d6c9245fb909fd6eac252a527ffe67173c30a94fcae75b22ce712becff5d"} Oct 31 13:56:34.397795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.397771 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:56:34.472472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.472433 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn"] Oct 31 13:56:34.485415 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.485377 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-4f340c-v52mn"] Oct 31 13:56:34.500371 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.500341 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-maas-api-tls\") pod \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " Oct 31 13:56:34.500516 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.500439 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-metrics-certs\") pod \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " Oct 31 13:56:34.500516 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.500490 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-openshift-service-ca\") pod \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " Oct 31 13:56:34.500776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.500754 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-kube-api-access-gs7w7\" (UniqueName: \"kubernetes.io/projected/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-kube-api-access-gs7w7\") pod \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\" (UID: \"aed42dc6-80f5-4a58-9b3d-ffca597ddef5\") " Oct 31 13:56:34.500907 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.500885 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-openshift-service-ca" pod "aed42dc6-80f5-4a58-9b3d-ffca597ddef5" (UID: "aed42dc6-80f5-4a58-9b3d-ffca597ddef5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:34.501460 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.501433 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:34.502568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.502543 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-maas-api-tls" pod "aed42dc6-80f5-4a58-9b3d-ffca597ddef5" (UID: "aed42dc6-80f5-4a58-9b3d-ffca597ddef5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:56:34.502813 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.502793 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-kube-api-access-gs7w7" pod "aed42dc6-80f5-4a58-9b3d-ffca597ddef5" (UID: "aed42dc6-80f5-4a58-9b3d-ffca597ddef5"). InnerVolumeSpecName "kube-api-access-gs7w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:34.502870 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.502827 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-metrics-certs" pod "aed42dc6-80f5-4a58-9b3d-ffca597ddef5" (UID: "aed42dc6-80f5-4a58-9b3d-ffca597ddef5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:56:34.603753 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.603718 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:34.603753 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.603750 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:34.603974 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:34.603763 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs7w7\" (UniqueName: \"kubernetes.io/projected/aed42dc6-80f5-4a58-9b3d-ffca597ddef5-kube-api-access-gs7w7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:35.378820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:35.378788 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" Oct 31 13:56:35.379254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:35.378790 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" event={"ID":"aed42dc6-80f5-4a58-9b3d-ffca597ddef5","Type":"ContainerDied","Data":"3e60a61ce28b648f0ed26c35cd0a88742e30f22240e2def3cbae2f678692874d"} Oct 31 13:56:35.379254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:35.378927 5532 scope.go:122] "RemoveContainer" containerID="e870d6c9245fb909fd6eac252a527ffe67173c30a94fcae75b22ce712becff5d" Oct 31 13:56:35.409259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:35.409227 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl"] Oct 31 13:56:35.415852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:35.415816 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl"] Oct 31 13:56:36.017152 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:36.017118 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9457b600-c4fd-4784-ac44-c348292258c4" path="/var/lib/kubelet/pods/9457b600-c4fd-4784-ac44-c348292258c4/volumes" Oct 31 13:56:36.017789 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:36.017769 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed42dc6-80f5-4a58-9b3d-ffca597ddef5" path="/var/lib/kubelet/pods/aed42dc6-80f5-4a58-9b3d-ffca597ddef5/volumes" Oct 31 13:56:37.340175 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:37.340137 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-4f340c-5d7f9d7c86-vn6rl" podUID="aed42dc6-80f5-4a58-9b3d-ffca597ddef5" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.129:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 13:56:37.898659 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:37.898588 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw_b4b2679c-6948-4251-aec3-cdd30eb68497/payload-processing/0.log" Oct 31 13:56:38.641288 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.641255 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6"] Oct 31 13:56:38.642116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.642099 5532 cpu_manager.go:414] "containerMap: removing container" podUID="01c8f8ae-be64-4501-adb5-dc2689f938bf" containerName="payload-processing" Oct 31 13:56:38.642179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.642118 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01c8f8ae-be64-4501-adb5-dc2689f938bf" containerName="payload-processing" Oct 31 13:56:38.642179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.642135 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2cce49b5-22bd-4655-90eb-75b5fb6cae5b" containerName="payload-pre-processing" Oct 31 13:56:38.642179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.642141 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cce49b5-22bd-4655-90eb-75b5fb6cae5b" containerName="payload-pre-processing" Oct 31 13:56:38.642179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.642159 5532 cpu_manager.go:414] "containerMap: removing container" podUID="aed42dc6-80f5-4a58-9b3d-ffca597ddef5" containerName="maas-api" Oct 31 13:56:38.642179 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.642164 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed42dc6-80f5-4a58-9b3d-ffca597ddef5" containerName="maas-api" Oct 31 13:56:38.642355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.642310 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed42dc6-80f5-4a58-9b3d-ffca597ddef5" containerName="maas-api" Oct 31 13:56:38.642355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.642330 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="01c8f8ae-be64-4501-adb5-dc2689f938bf" containerName="payload-processing" Oct 31 13:56:38.642355 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.642336 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cce49b5-22bd-4655-90eb-75b5fb6cae5b" containerName="payload-pre-processing" Oct 31 13:56:38.647187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.647168 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" Oct 31 13:56:38.653776 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.653745 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:56:38.653935 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.653869 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:56:38.654983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.654959 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6"] Oct 31 13:56:38.661234 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.661207 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8cm9\" (UniqueName: \"kubernetes.io/projected/bb16a781-3b5e-42f1-ac83-f7c241458398-kube-api-access-r8cm9\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6\" (UID: \"bb16a781-3b5e-42f1-ac83-f7c241458398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" Oct 31 13:56:38.661438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.661415 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6\" (UID: \"bb16a781-3b5e-42f1-ac83-f7c241458398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" Oct 31 13:56:38.661699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.661677 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6\" (UID: \"bb16a781-3b5e-42f1-ac83-f7c241458398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" Oct 31 13:56:38.763212 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.763172 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8cm9\" (UniqueName: \"kubernetes.io/projected/bb16a781-3b5e-42f1-ac83-f7c241458398-kube-api-access-r8cm9\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6\" (UID: \"bb16a781-3b5e-42f1-ac83-f7c241458398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" Oct 31 13:56:38.763405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.763290 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6\" (UID: \"bb16a781-3b5e-42f1-ac83-f7c241458398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" Oct 31 13:56:38.763472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.763454 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6\" (UID: \"bb16a781-3b5e-42f1-ac83-f7c241458398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" Oct 31 13:56:38.764014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.763989 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6\" (UID: \"bb16a781-3b5e-42f1-ac83-f7c241458398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" Oct 31 13:56:38.765882 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.765857 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6\" (UID: \"bb16a781-3b5e-42f1-ac83-f7c241458398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" Oct 31 13:56:38.777111 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.777083 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8cm9\" (UniqueName: \"kubernetes.io/projected/bb16a781-3b5e-42f1-ac83-f7c241458398-kube-api-access-r8cm9\") pod \"maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6\" (UID: \"bb16a781-3b5e-42f1-ac83-f7c241458398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" Oct 31 13:56:38.965082 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:38.965054 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" Oct 31 13:56:39.255622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:39.255154 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6"] Oct 31 13:56:39.257413 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:56:39.257377 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb16a781_3b5e_42f1_ac83_f7c241458398.slice/crio-65973c19c1a8b87c554aa74d401834a596df516027a98718936817b110e144fa WatchSource:0}: Error finding container 65973c19c1a8b87c554aa74d401834a596df516027a98718936817b110e144fa: Status 404 returned error can't find the container with id 65973c19c1a8b87c554aa74d401834a596df516027a98718936817b110e144fa Oct 31 13:56:39.424447 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:39.424413 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" event={"ID":"bb16a781-3b5e-42f1-ac83-f7c241458398","Type":"ContainerStarted","Data":"65973c19c1a8b87c554aa74d401834a596df516027a98718936817b110e144fa"} Oct 31 13:56:40.449387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:40.449341 5532 generic.go:356] "Generic (PLEG): container finished" podID="bb16a781-3b5e-42f1-ac83-f7c241458398" containerID="e381d8f8ba184d66db3440dc15d5e73c0f2291ccc2326aafc1cbe8d044cfb17b" exitCode=0 Oct 31 13:56:40.449911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:40.449396 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" event={"ID":"bb16a781-3b5e-42f1-ac83-f7c241458398","Type":"ContainerDied","Data":"e381d8f8ba184d66db3440dc15d5e73c0f2291ccc2326aafc1cbe8d044cfb17b"} Oct 31 13:56:41.647425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:41.647400 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" Oct 31 13:56:41.742347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:41.742255 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb16a781-3b5e-42f1-ac83-f7c241458398-kube-api-access-r8cm9\" (UniqueName: \"kubernetes.io/projected/bb16a781-3b5e-42f1-ac83-f7c241458398-kube-api-access-r8cm9\") pod \"bb16a781-3b5e-42f1-ac83-f7c241458398\" (UID: \"bb16a781-3b5e-42f1-ac83-f7c241458398\") " Oct 31 13:56:41.742347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:41.742339 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-openshift-ca\") pod \"bb16a781-3b5e-42f1-ac83-f7c241458398\" (UID: \"bb16a781-3b5e-42f1-ac83-f7c241458398\") " Oct 31 13:56:41.742625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:41.742449 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-xks-ca\") pod \"bb16a781-3b5e-42f1-ac83-f7c241458398\" (UID: \"bb16a781-3b5e-42f1-ac83-f7c241458398\") " Oct 31 13:56:41.742748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:41.742713 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-openshift-ca" pod "bb16a781-3b5e-42f1-ac83-f7c241458398" (UID: "bb16a781-3b5e-42f1-ac83-f7c241458398"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:41.743330 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:41.743301 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:41.744632 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:41.744586 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb16a781-3b5e-42f1-ac83-f7c241458398-kube-api-access-r8cm9" pod "bb16a781-3b5e-42f1-ac83-f7c241458398" (UID: "bb16a781-3b5e-42f1-ac83-f7c241458398"). InnerVolumeSpecName "kube-api-access-r8cm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:41.745007 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:41.744982 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-xks-ca" pod "bb16a781-3b5e-42f1-ac83-f7c241458398" (UID: "bb16a781-3b5e-42f1-ac83-f7c241458398"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:56:41.844752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:41.844711 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8cm9\" (UniqueName: \"kubernetes.io/projected/bb16a781-3b5e-42f1-ac83-f7c241458398-kube-api-access-r8cm9\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:41.844752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:41.844747 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb16a781-3b5e-42f1-ac83-f7c241458398-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:42.147930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.147005 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z"] Oct 31 13:56:42.147930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.147247 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerName="istio-proxy" containerID="cri-o://31773840c162b2e52d89cfea61fdfb82a72d67685ef86fc52f06ef4b91311f10" gracePeriod=30 Oct 31 13:56:42.470178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.470092 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" Oct 31 13:56:42.470178 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.470126 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6" event={"ID":"bb16a781-3b5e-42f1-ac83-f7c241458398","Type":"ContainerDied","Data":"65973c19c1a8b87c554aa74d401834a596df516027a98718936817b110e144fa"} Oct 31 13:56:42.470387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.470189 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65973c19c1a8b87c554aa74d401834a596df516027a98718936817b110e144fa" Oct 31 13:56:42.583851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.583820 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7"] Oct 31 13:56:42.584686 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.584670 5532 cpu_manager.go:414] "containerMap: removing container" podUID="bb16a781-3b5e-42f1-ac83-f7c241458398" containerName="revoke-keys" Oct 31 13:56:42.584747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.584688 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb16a781-3b5e-42f1-ac83-f7c241458398" containerName="revoke-keys" Oct 31 13:56:42.584854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.584844 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb16a781-3b5e-42f1-ac83-f7c241458398" containerName="revoke-keys" Oct 31 13:56:42.588889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.588872 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" Oct 31 13:56:42.597891 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.597655 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7"] Oct 31 13:56:42.642695 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.642668 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:56:42.643006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.642990 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:56:42.662078 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.662044 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7\" (UID: \"b224c68a-a55e-4485-a218-1fbdb9ab0437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" Oct 31 13:56:42.662447 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.662253 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7\" (UID: \"b224c68a-a55e-4485-a218-1fbdb9ab0437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" Oct 31 13:56:42.662447 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.662355 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79wr2\" (UniqueName: \"kubernetes.io/projected/b224c68a-a55e-4485-a218-1fbdb9ab0437-kube-api-access-79wr2\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7\" (UID: \"b224c68a-a55e-4485-a218-1fbdb9ab0437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" Oct 31 13:56:42.764209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.764128 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7\" (UID: \"b224c68a-a55e-4485-a218-1fbdb9ab0437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" Oct 31 13:56:42.764379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.764230 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79wr2\" (UniqueName: \"kubernetes.io/projected/b224c68a-a55e-4485-a218-1fbdb9ab0437-kube-api-access-79wr2\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7\" (UID: \"b224c68a-a55e-4485-a218-1fbdb9ab0437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" Oct 31 13:56:42.764379 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.764342 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7\" (UID: \"b224c68a-a55e-4485-a218-1fbdb9ab0437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" Oct 31 13:56:42.765044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.765015 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7\" (UID: \"b224c68a-a55e-4485-a218-1fbdb9ab0437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" Oct 31 13:56:42.766581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.766557 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7\" (UID: \"b224c68a-a55e-4485-a218-1fbdb9ab0437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" Oct 31 13:56:42.778726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.778701 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79wr2\" (UniqueName: \"kubernetes.io/projected/b224c68a-a55e-4485-a218-1fbdb9ab0437-kube-api-access-79wr2\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7\" (UID: \"b224c68a-a55e-4485-a218-1fbdb9ab0437\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" Oct 31 13:56:42.905748 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:42.905711 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" Oct 31 13:56:43.204320 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:56:43.204285 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb224c68a_a55e_4485_a218_1fbdb9ab0437.slice/crio-db2cfdebde853a1cb05f321a12c058052aa07c90a5f4f647eafae62144f2a586 WatchSource:0}: Error finding container db2cfdebde853a1cb05f321a12c058052aa07c90a5f4f647eafae62144f2a586: Status 404 returned error can't find the container with id db2cfdebde853a1cb05f321a12c058052aa07c90a5f4f647eafae62144f2a586 Oct 31 13:56:43.204478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:43.204459 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7"] Oct 31 13:56:43.480309 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:43.480272 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" event={"ID":"b224c68a-a55e-4485-a218-1fbdb9ab0437","Type":"ContainerStarted","Data":"db2cfdebde853a1cb05f321a12c058052aa07c90a5f4f647eafae62144f2a586"} Oct 31 13:56:44.490634 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:44.490502 5532 generic.go:356] "Generic (PLEG): container finished" podID="b224c68a-a55e-4485-a218-1fbdb9ab0437" containerID="a5cd7b8f43f685bcf846d023bc843cdb0581cd020ec1f68d1938410684e3ebbf" exitCode=0 Oct 31 13:56:44.491098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:44.490628 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" event={"ID":"b224c68a-a55e-4485-a218-1fbdb9ab0437","Type":"ContainerDied","Data":"a5cd7b8f43f685bcf846d023bc843cdb0581cd020ec1f68d1938410684e3ebbf"} Oct 31 13:56:45.683851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:45.683826 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" Oct 31 13:56:45.820346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:45.820265 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-openshift-ca\") pod \"b224c68a-a55e-4485-a218-1fbdb9ab0437\" (UID: \"b224c68a-a55e-4485-a218-1fbdb9ab0437\") " Oct 31 13:56:45.820499 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:45.820348 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b224c68a-a55e-4485-a218-1fbdb9ab0437-kube-api-access-79wr2\" (UniqueName: \"kubernetes.io/projected/b224c68a-a55e-4485-a218-1fbdb9ab0437-kube-api-access-79wr2\") pod \"b224c68a-a55e-4485-a218-1fbdb9ab0437\" (UID: \"b224c68a-a55e-4485-a218-1fbdb9ab0437\") " Oct 31 13:56:45.820499 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:45.820476 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-xks-ca\") pod \"b224c68a-a55e-4485-a218-1fbdb9ab0437\" (UID: \"b224c68a-a55e-4485-a218-1fbdb9ab0437\") " Oct 31 13:56:45.820682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:45.820653 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-openshift-ca" pod "b224c68a-a55e-4485-a218-1fbdb9ab0437" (UID: "b224c68a-a55e-4485-a218-1fbdb9ab0437"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:45.821488 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:45.821466 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:45.822509 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:45.822489 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b224c68a-a55e-4485-a218-1fbdb9ab0437-kube-api-access-79wr2" pod "b224c68a-a55e-4485-a218-1fbdb9ab0437" (UID: "b224c68a-a55e-4485-a218-1fbdb9ab0437"). InnerVolumeSpecName "kube-api-access-79wr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:45.822714 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:45.822695 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-xks-ca" pod "b224c68a-a55e-4485-a218-1fbdb9ab0437" (UID: "b224c68a-a55e-4485-a218-1fbdb9ab0437"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:56:45.923523 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:45.923489 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79wr2\" (UniqueName: \"kubernetes.io/projected/b224c68a-a55e-4485-a218-1fbdb9ab0437-kube-api-access-79wr2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:45.923523 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:45.923522 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b224c68a-a55e-4485-a218-1fbdb9ab0437-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:46.509864 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:46.509835 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" Oct 31 13:56:46.510046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:46.509862 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7" event={"ID":"b224c68a-a55e-4485-a218-1fbdb9ab0437","Type":"ContainerDied","Data":"db2cfdebde853a1cb05f321a12c058052aa07c90a5f4f647eafae62144f2a586"} Oct 31 13:56:46.510046 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:46.509911 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db2cfdebde853a1cb05f321a12c058052aa07c90a5f4f647eafae62144f2a586" Oct 31 13:56:47.521437 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.521356 5532 generic.go:356] "Generic (PLEG): container finished" podID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerID="31773840c162b2e52d89cfea61fdfb82a72d67685ef86fc52f06ef4b91311f10" exitCode=0 Oct 31 13:56:47.521906 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.521428 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" event={"ID":"a3dd044e-3702-46d9-91db-2d456c9dd345","Type":"ContainerDied","Data":"31773840c162b2e52d89cfea61fdfb82a72d67685ef86fc52f06ef4b91311f10"} Oct 31 13:56:47.521906 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.521489 5532 scope.go:122] "RemoveContainer" containerID="aac1668030455d43982c62c2232c93ad7105a4b718e8b8430cd35d2636767b9b" Oct 31 13:56:47.860979 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.860942 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:56:47.958080 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958050 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-socket\") pod \"a3dd044e-3702-46d9-91db-2d456c9dd345\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " Oct 31 13:56:47.958287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958107 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3dd044e-3702-46d9-91db-2d456c9dd345-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dd044e-3702-46d9-91db-2d456c9dd345-istiod-ca-cert\") pod \"a3dd044e-3702-46d9-91db-2d456c9dd345\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " Oct 31 13:56:47.958287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958127 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-token\" (UniqueName: \"kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-token\") pod \"a3dd044e-3702-46d9-91db-2d456c9dd345\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " Oct 31 13:56:47.958287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958230 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-credential-socket\") pod \"a3dd044e-3702-46d9-91db-2d456c9dd345\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " Oct 31 13:56:47.958287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958284 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-certs\") pod \"a3dd044e-3702-46d9-91db-2d456c9dd345\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " Oct 31 13:56:47.958533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958345 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-data\") pod \"a3dd044e-3702-46d9-91db-2d456c9dd345\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " Oct 31 13:56:47.958533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958351 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-socket" pod "a3dd044e-3702-46d9-91db-2d456c9dd345" (UID: "a3dd044e-3702-46d9-91db-2d456c9dd345"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:56:47.958533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958378 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-kube-api-access-8b7tl\" (UniqueName: \"kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-kube-api-access-8b7tl\") pod \"a3dd044e-3702-46d9-91db-2d456c9dd345\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " Oct 31 13:56:47.958533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958427 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-podinfo\") pod \"a3dd044e-3702-46d9-91db-2d456c9dd345\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " Oct 31 13:56:47.958820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958553 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-envoy\") pod \"a3dd044e-3702-46d9-91db-2d456c9dd345\" (UID: \"a3dd044e-3702-46d9-91db-2d456c9dd345\") " Oct 31 13:56:47.958820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958651 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-credential-socket" pod "a3dd044e-3702-46d9-91db-2d456c9dd345" (UID: "a3dd044e-3702-46d9-91db-2d456c9dd345"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:56:47.958820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958705 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-certs" pod "a3dd044e-3702-46d9-91db-2d456c9dd345" (UID: "a3dd044e-3702-46d9-91db-2d456c9dd345"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:56:47.958820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958694 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3dd044e-3702-46d9-91db-2d456c9dd345-istiod-ca-cert" pod "a3dd044e-3702-46d9-91db-2d456c9dd345" (UID: "a3dd044e-3702-46d9-91db-2d456c9dd345"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:47.958820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.958737 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-data" pod "a3dd044e-3702-46d9-91db-2d456c9dd345" (UID: "a3dd044e-3702-46d9-91db-2d456c9dd345"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:56:47.959348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.959324 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:47.959491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.959471 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dd044e-3702-46d9-91db-2d456c9dd345-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:47.959491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.959489 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:47.959712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.959503 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:47.959712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.959514 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:47.960677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.960657 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-envoy" pod "a3dd044e-3702-46d9-91db-2d456c9dd345" (UID: "a3dd044e-3702-46d9-91db-2d456c9dd345"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:56:47.960833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.960810 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-kube-api-access-8b7tl" pod "a3dd044e-3702-46d9-91db-2d456c9dd345" (UID: "a3dd044e-3702-46d9-91db-2d456c9dd345"). InnerVolumeSpecName "kube-api-access-8b7tl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:47.960945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.960902 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-token" pod "a3dd044e-3702-46d9-91db-2d456c9dd345" (UID: "a3dd044e-3702-46d9-91db-2d456c9dd345"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:47.961167 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:47.961150 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-podinfo" pod "a3dd044e-3702-46d9-91db-2d456c9dd345" (UID: "a3dd044e-3702-46d9-91db-2d456c9dd345"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:56:48.061897 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:48.061859 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b7tl\" (UniqueName: \"kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-kube-api-access-8b7tl\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:48.062169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:48.062045 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:48.062169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:48.062066 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:48.062169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:48.062075 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dd044e-3702-46d9-91db-2d456c9dd345-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:48.531286 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:48.531255 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" Oct 31 13:56:48.531747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:48.531255 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z" event={"ID":"a3dd044e-3702-46d9-91db-2d456c9dd345","Type":"ContainerDied","Data":"1a56626c2589590e0a0f86eeddb40c695fbc565d211db208a1b7b90ec9f44a5d"} Oct 31 13:56:48.531747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:48.531422 5532 scope.go:122] "RemoveContainer" containerID="31773840c162b2e52d89cfea61fdfb82a72d67685ef86fc52f06ef4b91311f10" Oct 31 13:56:48.560120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:48.560088 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z"] Oct 31 13:56:48.568895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:48.568868 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-4f340c-openshift-default-7bb446c46c-scg7z"] Oct 31 13:56:50.005976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:50.005944 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" path="/var/lib/kubelet/pods/a3dd044e-3702-46d9-91db-2d456c9dd345/volumes" Oct 31 13:56:52.064507 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.064476 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2"] Oct 31 13:56:52.065063 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.064690 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" podUID="1782c792-12e9-4104-bbc4-18108c4a6e7d" containerName="payload-processing" containerID="cri-o://b7c8dd3eb889869571e512ed5979940049d9a2745f4886479b41d85778edcb22" gracePeriod=30 Oct 31 13:56:52.086230 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.086206 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2"] Oct 31 13:56:52.086660 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.086628 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" podUID="2adbd9b5-1070-435e-81b4-f8846f493949" containerName="payload-pre-processing" containerID="cri-o://cc64e9cc2ec2c8440f09fe195da298558f457fae7082d59252e2bd6f5374bd2b" gracePeriod=30 Oct 31 13:56:52.578743 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.578706 5532 generic.go:356] "Generic (PLEG): container finished" podID="2adbd9b5-1070-435e-81b4-f8846f493949" containerID="cc64e9cc2ec2c8440f09fe195da298558f457fae7082d59252e2bd6f5374bd2b" exitCode=0 Oct 31 13:56:52.579284 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.579021 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" event={"ID":"2adbd9b5-1070-435e-81b4-f8846f493949","Type":"ContainerDied","Data":"cc64e9cc2ec2c8440f09fe195da298558f457fae7082d59252e2bd6f5374bd2b"} Oct 31 13:56:52.581922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.581750 5532 generic.go:356] "Generic (PLEG): container finished" podID="1782c792-12e9-4104-bbc4-18108c4a6e7d" containerID="b7c8dd3eb889869571e512ed5979940049d9a2745f4886479b41d85778edcb22" exitCode=0 Oct 31 13:56:52.581922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.581802 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" event={"ID":"1782c792-12e9-4104-bbc4-18108c4a6e7d","Type":"ContainerDied","Data":"b7c8dd3eb889869571e512ed5979940049d9a2745f4886479b41d85778edcb22"} Oct 31 13:56:52.581922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.581842 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" event={"ID":"1782c792-12e9-4104-bbc4-18108c4a6e7d","Type":"ContainerDied","Data":"f91559367b5b2d8c194f5257e6935a68a815a333734e4db241d285b99de1ab6c"} Oct 31 13:56:52.581922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.581869 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f91559367b5b2d8c194f5257e6935a68a815a333734e4db241d285b99de1ab6c" Oct 31 13:56:52.615373 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.614952 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" Oct 31 13:56:52.645183 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.645149 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1782c792-12e9-4104-bbc4-18108c4a6e7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1782c792-12e9-4104-bbc4-18108c4a6e7d-plugins-config-volume\") pod \"1782c792-12e9-4104-bbc4-18108c4a6e7d\" (UID: \"1782c792-12e9-4104-bbc4-18108c4a6e7d\") " Oct 31 13:56:52.647121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.647048 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1782c792-12e9-4104-bbc4-18108c4a6e7d-plugins-config-volume" pod "1782c792-12e9-4104-bbc4-18108c4a6e7d" (UID: "1782c792-12e9-4104-bbc4-18108c4a6e7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:52.648077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.647510 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1782c792-12e9-4104-bbc4-18108c4a6e7d-kube-api-access-6bjg9\" (UniqueName: \"kubernetes.io/projected/1782c792-12e9-4104-bbc4-18108c4a6e7d-kube-api-access-6bjg9\") pod \"1782c792-12e9-4104-bbc4-18108c4a6e7d\" (UID: \"1782c792-12e9-4104-bbc4-18108c4a6e7d\") " Oct 31 13:56:52.651203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.651184 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1782c792-12e9-4104-bbc4-18108c4a6e7d-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:52.667054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.667007 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1782c792-12e9-4104-bbc4-18108c4a6e7d-kube-api-access-6bjg9" pod "1782c792-12e9-4104-bbc4-18108c4a6e7d" (UID: "1782c792-12e9-4104-bbc4-18108c4a6e7d"). InnerVolumeSpecName "kube-api-access-6bjg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:52.710934 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.710910 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" Oct 31 13:56:52.752446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.752409 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6"] Oct 31 13:56:52.753192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.753162 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2adbd9b5-1070-435e-81b4-f8846f493949-kube-api-access-nd2wm\" (UniqueName: \"kubernetes.io/projected/2adbd9b5-1070-435e-81b4-f8846f493949-kube-api-access-nd2wm\") pod \"2adbd9b5-1070-435e-81b4-f8846f493949\" (UID: \"2adbd9b5-1070-435e-81b4-f8846f493949\") " Oct 31 13:56:52.753457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.753441 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2adbd9b5-1070-435e-81b4-f8846f493949-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2adbd9b5-1070-435e-81b4-f8846f493949-plugins-config-volume\") pod \"2adbd9b5-1070-435e-81b4-f8846f493949\" (UID: \"2adbd9b5-1070-435e-81b4-f8846f493949\") " Oct 31 13:56:52.753957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.753942 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bjg9\" (UniqueName: \"kubernetes.io/projected/1782c792-12e9-4104-bbc4-18108c4a6e7d-kube-api-access-6bjg9\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:52.754274 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.754248 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2adbd9b5-1070-435e-81b4-f8846f493949-plugins-config-volume" pod "2adbd9b5-1070-435e-81b4-f8846f493949" (UID: "2adbd9b5-1070-435e-81b4-f8846f493949"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:52.756521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.756492 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2adbd9b5-1070-435e-81b4-f8846f493949-kube-api-access-nd2wm" pod "2adbd9b5-1070-435e-81b4-f8846f493949" (UID: "2adbd9b5-1070-435e-81b4-f8846f493949"). InnerVolumeSpecName "kube-api-access-nd2wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:52.759988 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.759963 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4be675-qxxc6"] Oct 31 13:56:52.813995 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.813925 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6"] Oct 31 13:56:52.834749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.834720 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4"] Oct 31 13:56:52.856674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.856639 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd2wm\" (UniqueName: \"kubernetes.io/projected/2adbd9b5-1070-435e-81b4-f8846f493949-kube-api-access-nd2wm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:52.856674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:52.856671 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2adbd9b5-1070-435e-81b4-f8846f493949-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:53.592086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:53.592058 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" Oct 31 13:56:53.592554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:53.592099 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2" event={"ID":"2adbd9b5-1070-435e-81b4-f8846f493949","Type":"ContainerDied","Data":"5411f3a36b1e49d26c0df6d3f87bee3f231f3be9dbc6c0dc0d79cb4778115968"} Oct 31 13:56:53.592554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:53.592156 5532 scope.go:122] "RemoveContainer" containerID="cc64e9cc2ec2c8440f09fe195da298558f457fae7082d59252e2bd6f5374bd2b" Oct 31 13:56:53.592554 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:53.592372 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2" Oct 31 13:56:53.626226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:53.626203 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2"] Oct 31 13:56:53.632316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:53.632276 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4be675-846c64f56c-fwnb2"] Oct 31 13:56:53.643743 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:53.643700 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2"] Oct 31 13:56:53.649448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:53.649424 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4be675-85cb959c76-r8sz2"] Oct 31 13:56:54.008409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:54.008369 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1782c792-12e9-4104-bbc4-18108c4a6e7d" path="/var/lib/kubelet/pods/1782c792-12e9-4104-bbc4-18108c4a6e7d/volumes" Oct 31 13:56:54.009141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:54.009118 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2adbd9b5-1070-435e-81b4-f8846f493949" path="/var/lib/kubelet/pods/2adbd9b5-1070-435e-81b4-f8846f493949/volumes" Oct 31 13:56:54.009795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:54.009778 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb16a781-3b5e-42f1-ac83-f7c241458398" path="/var/lib/kubelet/pods/bb16a781-3b5e-42f1-ac83-f7c241458398/volumes" Oct 31 13:56:55.923043 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:55.923011 5532 scope.go:122] "RemoveContainer" containerID="0206082d7ef3b4aebcdcdde880648e37c3d18b1f19263e9246bff8b1e308a879" Oct 31 13:56:57.732452 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:57.732420 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:56:57.815102 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:57.815066 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" podUID="6cc022d9-8637-426b-b8fe-6eeafb3bf823" containerName="payload-processing" containerID="cri-o://b1df64ab6cc1fa008391dda524cad636852b4e3facca3f42d712f2f67a3cfb51" gracePeriod=25 Oct 31 13:56:57.835635 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:57.835575 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" podUID="614f22b9-7512-481c-b8e3-f4757bca9265" containerName="payload-pre-processing" containerID="cri-o://2ac91c2461f838df11f47a1041d4ef4e44e25769e87f07bbc0aca267546c4f85" gracePeriod=25 Oct 31 13:56:58.184462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.184432 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" Oct 31 13:56:58.205913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.205747 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" Oct 31 13:56:58.249346 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.249313 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc022d9-8637-426b-b8fe-6eeafb3bf823-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc022d9-8637-426b-b8fe-6eeafb3bf823-plugins-config-volume\") pod \"6cc022d9-8637-426b-b8fe-6eeafb3bf823\" (UID: \"6cc022d9-8637-426b-b8fe-6eeafb3bf823\") " Oct 31 13:56:58.249545 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.249363 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc022d9-8637-426b-b8fe-6eeafb3bf823-kube-api-access-mrtfv\" (UniqueName: \"kubernetes.io/projected/6cc022d9-8637-426b-b8fe-6eeafb3bf823-kube-api-access-mrtfv\") pod \"6cc022d9-8637-426b-b8fe-6eeafb3bf823\" (UID: \"6cc022d9-8637-426b-b8fe-6eeafb3bf823\") " Oct 31 13:56:58.249759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.249733 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc022d9-8637-426b-b8fe-6eeafb3bf823-plugins-config-volume" pod "6cc022d9-8637-426b-b8fe-6eeafb3bf823" (UID: "6cc022d9-8637-426b-b8fe-6eeafb3bf823"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:58.250140 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.250122 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cc022d9-8637-426b-b8fe-6eeafb3bf823-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:58.251544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.251525 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc022d9-8637-426b-b8fe-6eeafb3bf823-kube-api-access-mrtfv" pod "6cc022d9-8637-426b-b8fe-6eeafb3bf823" (UID: "6cc022d9-8637-426b-b8fe-6eeafb3bf823"). InnerVolumeSpecName "kube-api-access-mrtfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:58.353780 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.353679 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/614f22b9-7512-481c-b8e3-f4757bca9265-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614f22b9-7512-481c-b8e3-f4757bca9265-plugins-config-volume\") pod \"614f22b9-7512-481c-b8e3-f4757bca9265\" (UID: \"614f22b9-7512-481c-b8e3-f4757bca9265\") " Oct 31 13:56:58.353972 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.353949 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/614f22b9-7512-481c-b8e3-f4757bca9265-kube-api-access-vvzlb\" (UniqueName: \"kubernetes.io/projected/614f22b9-7512-481c-b8e3-f4757bca9265-kube-api-access-vvzlb\") pod \"614f22b9-7512-481c-b8e3-f4757bca9265\" (UID: \"614f22b9-7512-481c-b8e3-f4757bca9265\") " Oct 31 13:56:58.354291 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.354260 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/614f22b9-7512-481c-b8e3-f4757bca9265-plugins-config-volume" pod "614f22b9-7512-481c-b8e3-f4757bca9265" (UID: "614f22b9-7512-481c-b8e3-f4757bca9265"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:56:58.356816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.356795 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrtfv\" (UniqueName: \"kubernetes.io/projected/6cc022d9-8637-426b-b8fe-6eeafb3bf823-kube-api-access-mrtfv\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:58.356816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.356816 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/614f22b9-7512-481c-b8e3-f4757bca9265-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:58.357454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.357427 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614f22b9-7512-481c-b8e3-f4757bca9265-kube-api-access-vvzlb" pod "614f22b9-7512-481c-b8e3-f4757bca9265" (UID: "614f22b9-7512-481c-b8e3-f4757bca9265"). InnerVolumeSpecName "kube-api-access-vvzlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:56:58.458291 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.458252 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvzlb\" (UniqueName: \"kubernetes.io/projected/614f22b9-7512-481c-b8e3-f4757bca9265-kube-api-access-vvzlb\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:56:58.731973 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.731940 5532 generic.go:356] "Generic (PLEG): container finished" podID="614f22b9-7512-481c-b8e3-f4757bca9265" containerID="2ac91c2461f838df11f47a1041d4ef4e44e25769e87f07bbc0aca267546c4f85" exitCode=0 Oct 31 13:56:58.732483 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.732284 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" Oct 31 13:56:58.734676 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.732325 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" event={"ID":"614f22b9-7512-481c-b8e3-f4757bca9265","Type":"ContainerDied","Data":"2ac91c2461f838df11f47a1041d4ef4e44e25769e87f07bbc0aca267546c4f85"} Oct 31 13:56:58.734867 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.734847 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4" event={"ID":"614f22b9-7512-481c-b8e3-f4757bca9265","Type":"ContainerDied","Data":"54d2be61934a0d85e371ac3fdf35cad57595661bb1f15f1e287c6eb49a84016e"} Oct 31 13:56:58.734995 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.734979 5532 scope.go:122] "RemoveContainer" containerID="2ac91c2461f838df11f47a1041d4ef4e44e25769e87f07bbc0aca267546c4f85" Oct 31 13:56:58.736425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.736403 5532 generic.go:356] "Generic (PLEG): container finished" podID="6cc022d9-8637-426b-b8fe-6eeafb3bf823" containerID="b1df64ab6cc1fa008391dda524cad636852b4e3facca3f42d712f2f67a3cfb51" exitCode=0 Oct 31 13:56:58.736809 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.736610 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" event={"ID":"6cc022d9-8637-426b-b8fe-6eeafb3bf823","Type":"ContainerDied","Data":"b1df64ab6cc1fa008391dda524cad636852b4e3facca3f42d712f2f67a3cfb51"} Oct 31 13:56:58.736809 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.736757 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" event={"ID":"6cc022d9-8637-426b-b8fe-6eeafb3bf823","Type":"ContainerDied","Data":"aeda4143e7502541419b7fd0c069c71b35bf7e1c55ff3401c567677f9c08b86f"} Oct 31 13:56:58.736809 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.736622 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6" Oct 31 13:56:58.767628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.767313 5532 scope.go:122] "RemoveContainer" containerID="2ac91c2461f838df11f47a1041d4ef4e44e25769e87f07bbc0aca267546c4f85" Oct 31 13:56:58.767867 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:56:58.767831 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ac91c2461f838df11f47a1041d4ef4e44e25769e87f07bbc0aca267546c4f85\": container with ID starting with 2ac91c2461f838df11f47a1041d4ef4e44e25769e87f07bbc0aca267546c4f85 not found: ID does not exist" containerID="2ac91c2461f838df11f47a1041d4ef4e44e25769e87f07bbc0aca267546c4f85" Oct 31 13:56:58.767991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.767868 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ac91c2461f838df11f47a1041d4ef4e44e25769e87f07bbc0aca267546c4f85"} err="failed to get container status \"2ac91c2461f838df11f47a1041d4ef4e44e25769e87f07bbc0aca267546c4f85\": rpc error: code = NotFound desc = could not find container \"2ac91c2461f838df11f47a1041d4ef4e44e25769e87f07bbc0aca267546c4f85\": container with ID starting with 2ac91c2461f838df11f47a1041d4ef4e44e25769e87f07bbc0aca267546c4f85 not found: ID does not exist" Oct 31 13:56:58.767991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.767894 5532 scope.go:122] "RemoveContainer" containerID="b1df64ab6cc1fa008391dda524cad636852b4e3facca3f42d712f2f67a3cfb51" Oct 31 13:56:58.781036 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.781003 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6"] Oct 31 13:56:58.782624 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.782272 5532 scope.go:122] "RemoveContainer" containerID="b1df64ab6cc1fa008391dda524cad636852b4e3facca3f42d712f2f67a3cfb51" Oct 31 13:56:58.782725 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:56:58.782652 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1df64ab6cc1fa008391dda524cad636852b4e3facca3f42d712f2f67a3cfb51\": container with ID starting with b1df64ab6cc1fa008391dda524cad636852b4e3facca3f42d712f2f67a3cfb51 not found: ID does not exist" containerID="b1df64ab6cc1fa008391dda524cad636852b4e3facca3f42d712f2f67a3cfb51" Oct 31 13:56:58.782725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.782683 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1df64ab6cc1fa008391dda524cad636852b4e3facca3f42d712f2f67a3cfb51"} err="failed to get container status \"b1df64ab6cc1fa008391dda524cad636852b4e3facca3f42d712f2f67a3cfb51\": rpc error: code = NotFound desc = could not find container \"b1df64ab6cc1fa008391dda524cad636852b4e3facca3f42d712f2f67a3cfb51\": container with ID starting with b1df64ab6cc1fa008391dda524cad636852b4e3facca3f42d712f2f67a3cfb51 not found: ID does not exist" Oct 31 13:56:58.787784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.787760 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-b06b08-6cb9f4844c-9mkm6"] Oct 31 13:56:58.806909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.806876 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4"] Oct 31 13:56:58.817923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:56:58.816935 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-b06b08-8688ccc55f-6r5d4"] Oct 31 13:57:00.005566 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.005525 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="614f22b9-7512-481c-b8e3-f4757bca9265" path="/var/lib/kubelet/pods/614f22b9-7512-481c-b8e3-f4757bca9265/volumes" Oct 31 13:57:00.006157 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.006138 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc022d9-8637-426b-b8fe-6eeafb3bf823" path="/var/lib/kubelet/pods/6cc022d9-8637-426b-b8fe-6eeafb3bf823/volumes" Oct 31 13:57:00.234628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.234554 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8"] Oct 31 13:57:00.234854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.234828 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" containerID="cri-o://b4e5e0874fbecabb974eb2b5cc9705f41c840dd35890a4e90c2d5542468c3230" gracePeriod=30 Oct 31 13:57:00.979135 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.979095 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb"] Oct 31 13:57:00.980363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980338 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerName="istio-proxy" Oct 31 13:57:00.980454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980368 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerName="istio-proxy" Oct 31 13:57:00.980454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980389 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerName="istio-proxy" Oct 31 13:57:00.980454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980399 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerName="istio-proxy" Oct 31 13:57:00.980454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980421 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2adbd9b5-1070-435e-81b4-f8846f493949" containerName="payload-pre-processing" Oct 31 13:57:00.980454 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980430 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2adbd9b5-1070-435e-81b4-f8846f493949" containerName="payload-pre-processing" Oct 31 13:57:00.980666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980452 5532 cpu_manager.go:414] "containerMap: removing container" podUID="6cc022d9-8637-426b-b8fe-6eeafb3bf823" containerName="payload-processing" Oct 31 13:57:00.980666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980463 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc022d9-8637-426b-b8fe-6eeafb3bf823" containerName="payload-processing" Oct 31 13:57:00.980666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980509 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b224c68a-a55e-4485-a218-1fbdb9ab0437" containerName="revoke-keys" Oct 31 13:57:00.980666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980517 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b224c68a-a55e-4485-a218-1fbdb9ab0437" containerName="revoke-keys" Oct 31 13:57:00.980666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980541 5532 cpu_manager.go:414] "containerMap: removing container" podUID="614f22b9-7512-481c-b8e3-f4757bca9265" containerName="payload-pre-processing" Oct 31 13:57:00.980666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980550 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="614f22b9-7512-481c-b8e3-f4757bca9265" containerName="payload-pre-processing" Oct 31 13:57:00.980666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980568 5532 cpu_manager.go:414] "containerMap: removing container" podUID="1782c792-12e9-4104-bbc4-18108c4a6e7d" containerName="payload-processing" Oct 31 13:57:00.980666 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980575 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1782c792-12e9-4104-bbc4-18108c4a6e7d" containerName="payload-processing" Oct 31 13:57:00.980912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980777 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerName="istio-proxy" Oct 31 13:57:00.980912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980795 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3dd044e-3702-46d9-91db-2d456c9dd345" containerName="istio-proxy" Oct 31 13:57:00.980912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980812 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2adbd9b5-1070-435e-81b4-f8846f493949" containerName="payload-pre-processing" Oct 31 13:57:00.980912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980823 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc022d9-8637-426b-b8fe-6eeafb3bf823" containerName="payload-processing" Oct 31 13:57:00.980912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980838 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b224c68a-a55e-4485-a218-1fbdb9ab0437" containerName="revoke-keys" Oct 31 13:57:00.980912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980851 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="614f22b9-7512-481c-b8e3-f4757bca9265" containerName="payload-pre-processing" Oct 31 13:57:00.980912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.980864 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="1782c792-12e9-4104-bbc4-18108c4a6e7d" containerName="payload-processing" Oct 31 13:57:00.986670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.986438 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" Oct 31 13:57:00.993275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.992758 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:57:00.993275 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.992910 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:57:00.994305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:00.994276 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb"] Oct 31 13:57:01.063271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:01.063237 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdcsr\" (UniqueName: \"kubernetes.io/projected/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-kube-api-access-fdcsr\") pod \"maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb\" (UID: \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" Oct 31 13:57:01.063777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:01.063480 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb\" (UID: \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" Oct 31 13:57:01.063902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:01.063875 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb\" (UID: \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" Oct 31 13:57:01.165377 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:01.165344 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb\" (UID: \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" Oct 31 13:57:01.165564 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:01.165531 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb\" (UID: \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" Oct 31 13:57:01.165702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:01.165685 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdcsr\" (UniqueName: \"kubernetes.io/projected/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-kube-api-access-fdcsr\") pod \"maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb\" (UID: \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" Oct 31 13:57:01.166035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:01.166003 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb\" (UID: \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" Oct 31 13:57:01.167985 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:01.167960 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb\" (UID: \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" Oct 31 13:57:01.180740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:01.180714 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdcsr\" (UniqueName: \"kubernetes.io/projected/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-kube-api-access-fdcsr\") pod \"maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb\" (UID: \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" Oct 31 13:57:01.318189 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:01.318095 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" Oct 31 13:57:01.611686 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:57:01.611653 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda83d7f4a_1ad3_4721_aaa6_69a7408d5697.slice/crio-058d8e0df90749f14ff875284d231ba098aa3c2504a00c904bfeb8338c129072 WatchSource:0}: Error finding container 058d8e0df90749f14ff875284d231ba098aa3c2504a00c904bfeb8338c129072: Status 404 returned error can't find the container with id 058d8e0df90749f14ff875284d231ba098aa3c2504a00c904bfeb8338c129072 Oct 31 13:57:01.612109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:01.612081 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb"] Oct 31 13:57:01.764365 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:57:01.764327 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:57:01.800276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:01.800231 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" event={"ID":"a83d7f4a-1ad3-4721-aaa6-69a7408d5697","Type":"ContainerStarted","Data":"058d8e0df90749f14ff875284d231ba098aa3c2504a00c904bfeb8338c129072"} Oct 31 13:57:02.813129 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:02.813089 5532 generic.go:356] "Generic (PLEG): container finished" podID="a83d7f4a-1ad3-4721-aaa6-69a7408d5697" containerID="f866c326c265207da9168507f6156338724e8b96762e8b9fb765e8200b215701" exitCode=0 Oct 31 13:57:02.813549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:02.813399 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" event={"ID":"a83d7f4a-1ad3-4721-aaa6-69a7408d5697","Type":"ContainerDied","Data":"f866c326c265207da9168507f6156338724e8b96762e8b9fb765e8200b215701"} Oct 31 13:57:03.998224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:03.998197 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" Oct 31 13:57:04.117689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:04.117546 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-kube-api-access-fdcsr\" (UniqueName: \"kubernetes.io/projected/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-kube-api-access-fdcsr\") pod \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\" (UID: \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\") " Oct 31 13:57:04.117689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:04.117609 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-openshift-ca\") pod \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\" (UID: \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\") " Oct 31 13:57:04.117689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:04.117672 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-xks-ca\") pod \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\" (UID: \"a83d7f4a-1ad3-4721-aaa6-69a7408d5697\") " Oct 31 13:57:04.117999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:04.117947 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-openshift-ca" pod "a83d7f4a-1ad3-4721-aaa6-69a7408d5697" (UID: "a83d7f4a-1ad3-4721-aaa6-69a7408d5697"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:04.118689 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:04.118664 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:04.119981 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:04.119955 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-kube-api-access-fdcsr" pod "a83d7f4a-1ad3-4721-aaa6-69a7408d5697" (UID: "a83d7f4a-1ad3-4721-aaa6-69a7408d5697"). InnerVolumeSpecName "kube-api-access-fdcsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:04.120070 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:04.120022 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-xks-ca" pod "a83d7f4a-1ad3-4721-aaa6-69a7408d5697" (UID: "a83d7f4a-1ad3-4721-aaa6-69a7408d5697"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:57:04.220258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:04.220223 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdcsr\" (UniqueName: \"kubernetes.io/projected/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-kube-api-access-fdcsr\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:04.220258 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:04.220255 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a83d7f4a-1ad3-4721-aaa6-69a7408d5697-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:04.832843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:04.832814 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" Oct 31 13:57:04.833031 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:04.832848 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb" event={"ID":"a83d7f4a-1ad3-4721-aaa6-69a7408d5697","Type":"ContainerDied","Data":"058d8e0df90749f14ff875284d231ba098aa3c2504a00c904bfeb8338c129072"} Oct 31 13:57:04.833031 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:04.832898 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="058d8e0df90749f14ff875284d231ba098aa3c2504a00c904bfeb8338c129072" Oct 31 13:57:05.553059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.553032 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:57:05.643574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.643528 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-kube-api-access-564lb\" (UniqueName: \"kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-kube-api-access-564lb\") pod \"4723d399-8244-4fd0-81f2-ce35a33441c8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " Oct 31 13:57:05.643832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.643605 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4723d399-8244-4fd0-81f2-ce35a33441c8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4723d399-8244-4fd0-81f2-ce35a33441c8-istiod-ca-cert\") pod \"4723d399-8244-4fd0-81f2-ce35a33441c8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " Oct 31 13:57:05.643832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.643690 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-token\" (UniqueName: \"kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-token\") pod \"4723d399-8244-4fd0-81f2-ce35a33441c8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " Oct 31 13:57:05.643832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.643724 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-socket\") pod \"4723d399-8244-4fd0-81f2-ce35a33441c8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " Oct 31 13:57:05.643832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.643766 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-certs\") pod \"4723d399-8244-4fd0-81f2-ce35a33441c8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " Oct 31 13:57:05.643832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.643805 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-envoy\") pod \"4723d399-8244-4fd0-81f2-ce35a33441c8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " Oct 31 13:57:05.644132 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.643891 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-podinfo\") pod \"4723d399-8244-4fd0-81f2-ce35a33441c8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " Oct 31 13:57:05.644132 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.644093 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-data\") pod \"4723d399-8244-4fd0-81f2-ce35a33441c8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " Oct 31 13:57:05.644247 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.644166 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-credential-socket\") pod \"4723d399-8244-4fd0-81f2-ce35a33441c8\" (UID: \"4723d399-8244-4fd0-81f2-ce35a33441c8\") " Oct 31 13:57:05.645191 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.645165 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-credential-socket" pod "4723d399-8244-4fd0-81f2-ce35a33441c8" (UID: "4723d399-8244-4fd0-81f2-ce35a33441c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:57:05.645745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.645721 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-certs" pod "4723d399-8244-4fd0-81f2-ce35a33441c8" (UID: "4723d399-8244-4fd0-81f2-ce35a33441c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:57:05.645861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.645838 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-socket" pod "4723d399-8244-4fd0-81f2-ce35a33441c8" (UID: "4723d399-8244-4fd0-81f2-ce35a33441c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:57:05.646072 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.646050 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-data" pod "4723d399-8244-4fd0-81f2-ce35a33441c8" (UID: "4723d399-8244-4fd0-81f2-ce35a33441c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:57:05.646151 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.646134 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4723d399-8244-4fd0-81f2-ce35a33441c8-istiod-ca-cert" pod "4723d399-8244-4fd0-81f2-ce35a33441c8" (UID: "4723d399-8244-4fd0-81f2-ce35a33441c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:05.647928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.647886 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-kube-api-access-564lb" pod "4723d399-8244-4fd0-81f2-ce35a33441c8" (UID: "4723d399-8244-4fd0-81f2-ce35a33441c8"). InnerVolumeSpecName "kube-api-access-564lb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:05.648335 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.648311 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-podinfo" pod "4723d399-8244-4fd0-81f2-ce35a33441c8" (UID: "4723d399-8244-4fd0-81f2-ce35a33441c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:57:05.648969 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.648942 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-envoy" pod "4723d399-8244-4fd0-81f2-ce35a33441c8" (UID: "4723d399-8244-4fd0-81f2-ce35a33441c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:57:05.649069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.648991 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-token" pod "4723d399-8244-4fd0-81f2-ce35a33441c8" (UID: "4723d399-8244-4fd0-81f2-ce35a33441c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:05.746497 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.746465 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-564lb\" (UniqueName: \"kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-kube-api-access-564lb\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:05.746497 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.746492 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4723d399-8244-4fd0-81f2-ce35a33441c8-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:05.746497 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.746502 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:05.746866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.746512 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:05.746866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.746522 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:05.746866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.746530 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:05.746866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.746540 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:05.746866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.746548 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:05.746866 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.746556 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4723d399-8244-4fd0-81f2-ce35a33441c8-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:05.842709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.842617 5532 generic.go:356] "Generic (PLEG): container finished" podID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerID="b4e5e0874fbecabb974eb2b5cc9705f41c840dd35890a4e90c2d5542468c3230" exitCode=0 Oct 31 13:57:05.842709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.842653 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" event={"ID":"4723d399-8244-4fd0-81f2-ce35a33441c8","Type":"ContainerDied","Data":"b4e5e0874fbecabb974eb2b5cc9705f41c840dd35890a4e90c2d5542468c3230"} Oct 31 13:57:05.842709 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.842681 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" Oct 31 13:57:05.842976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.842712 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8" event={"ID":"4723d399-8244-4fd0-81f2-ce35a33441c8","Type":"ContainerDied","Data":"da0969eeaea313275d64295061d3872f167b7f39f689fa80a497a698dacdfa42"} Oct 31 13:57:05.842976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.842734 5532 scope.go:122] "RemoveContainer" containerID="b4e5e0874fbecabb974eb2b5cc9705f41c840dd35890a4e90c2d5542468c3230" Oct 31 13:57:05.855266 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.855249 5532 scope.go:122] "RemoveContainer" containerID="c79c50c5d0949535c4d1acb318fdcc07c4b14bfa3d7478b34271be8a1112662e" Oct 31 13:57:05.866539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.866498 5532 scope.go:122] "RemoveContainer" containerID="b4e5e0874fbecabb974eb2b5cc9705f41c840dd35890a4e90c2d5542468c3230" Oct 31 13:57:05.866963 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:57:05.866936 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4e5e0874fbecabb974eb2b5cc9705f41c840dd35890a4e90c2d5542468c3230\": container with ID starting with b4e5e0874fbecabb974eb2b5cc9705f41c840dd35890a4e90c2d5542468c3230 not found: ID does not exist" containerID="b4e5e0874fbecabb974eb2b5cc9705f41c840dd35890a4e90c2d5542468c3230" Oct 31 13:57:05.867077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.866971 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4e5e0874fbecabb974eb2b5cc9705f41c840dd35890a4e90c2d5542468c3230"} err="failed to get container status \"b4e5e0874fbecabb974eb2b5cc9705f41c840dd35890a4e90c2d5542468c3230\": rpc error: code = NotFound desc = could not find container \"b4e5e0874fbecabb974eb2b5cc9705f41c840dd35890a4e90c2d5542468c3230\": container with ID starting with b4e5e0874fbecabb974eb2b5cc9705f41c840dd35890a4e90c2d5542468c3230 not found: ID does not exist" Oct 31 13:57:05.867077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.866994 5532 scope.go:122] "RemoveContainer" containerID="c79c50c5d0949535c4d1acb318fdcc07c4b14bfa3d7478b34271be8a1112662e" Oct 31 13:57:05.867247 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:57:05.867230 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c79c50c5d0949535c4d1acb318fdcc07c4b14bfa3d7478b34271be8a1112662e\": container with ID starting with c79c50c5d0949535c4d1acb318fdcc07c4b14bfa3d7478b34271be8a1112662e not found: ID does not exist" containerID="c79c50c5d0949535c4d1acb318fdcc07c4b14bfa3d7478b34271be8a1112662e" Oct 31 13:57:05.867294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.867253 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c79c50c5d0949535c4d1acb318fdcc07c4b14bfa3d7478b34271be8a1112662e"} err="failed to get container status \"c79c50c5d0949535c4d1acb318fdcc07c4b14bfa3d7478b34271be8a1112662e\": rpc error: code = NotFound desc = could not find container \"c79c50c5d0949535c4d1acb318fdcc07c4b14bfa3d7478b34271be8a1112662e\": container with ID starting with c79c50c5d0949535c4d1acb318fdcc07c4b14bfa3d7478b34271be8a1112662e not found: ID does not exist" Oct 31 13:57:05.872568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.872548 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8"] Oct 31 13:57:05.882868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:05.882837 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4be675-openshift-default-6cddb4cb85-jwrd8"] Oct 31 13:57:06.123669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:06.121628 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" path="/var/lib/kubelet/pods/4723d399-8244-4fd0-81f2-ce35a33441c8/volumes" Oct 31 13:57:07.256763 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.256723 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd"] Oct 31 13:57:07.257729 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.257705 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a83d7f4a-1ad3-4721-aaa6-69a7408d5697" containerName="revoke-keys" Oct 31 13:57:07.257729 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.257726 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a83d7f4a-1ad3-4721-aaa6-69a7408d5697" containerName="revoke-keys" Oct 31 13:57:07.257929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.257773 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" Oct 31 13:57:07.257929 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.257781 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" Oct 31 13:57:07.258030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.258005 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" Oct 31 13:57:07.258030 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.258021 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a83d7f4a-1ad3-4721-aaa6-69a7408d5697" containerName="revoke-keys" Oct 31 13:57:07.258129 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.258037 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" Oct 31 13:57:07.263468 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.263445 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" Oct 31 13:57:07.270337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.270315 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:57:07.270453 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.270416 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:57:07.272219 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.272197 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd"] Oct 31 13:57:07.406016 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.405981 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5e85a70-zqknd\" (UID: \"56ae5236-c47d-4182-9d85-9b8131f11783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" Oct 31 13:57:07.406224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.406104 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmq9x\" (UniqueName: \"kubernetes.io/projected/56ae5236-c47d-4182-9d85-9b8131f11783-kube-api-access-bmq9x\") pod \"maas-api-revoke-sub-c5e85a70-zqknd\" (UID: \"56ae5236-c47d-4182-9d85-9b8131f11783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" Oct 31 13:57:07.406291 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.406251 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5e85a70-zqknd\" (UID: \"56ae5236-c47d-4182-9d85-9b8131f11783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" Oct 31 13:57:07.507720 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.507621 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5e85a70-zqknd\" (UID: \"56ae5236-c47d-4182-9d85-9b8131f11783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" Oct 31 13:57:07.507913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.507745 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmq9x\" (UniqueName: \"kubernetes.io/projected/56ae5236-c47d-4182-9d85-9b8131f11783-kube-api-access-bmq9x\") pod \"maas-api-revoke-sub-c5e85a70-zqknd\" (UID: \"56ae5236-c47d-4182-9d85-9b8131f11783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" Oct 31 13:57:07.507913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.507899 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5e85a70-zqknd\" (UID: \"56ae5236-c47d-4182-9d85-9b8131f11783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" Oct 31 13:57:07.508536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.508503 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5e85a70-zqknd\" (UID: \"56ae5236-c47d-4182-9d85-9b8131f11783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" Oct 31 13:57:07.510568 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.510534 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5e85a70-zqknd\" (UID: \"56ae5236-c47d-4182-9d85-9b8131f11783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" Oct 31 13:57:07.521787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.521760 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmq9x\" (UniqueName: \"kubernetes.io/projected/56ae5236-c47d-4182-9d85-9b8131f11783-kube-api-access-bmq9x\") pod \"maas-api-revoke-sub-c5e85a70-zqknd\" (UID: \"56ae5236-c47d-4182-9d85-9b8131f11783\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" Oct 31 13:57:07.579989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.579958 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" Oct 31 13:57:07.868303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:07.868265 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd"] Oct 31 13:57:07.869490 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:57:07.869457 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ae5236_c47d_4182_9d85_9b8131f11783.slice/crio-f67803051cfc98fcb874b56cf6ffc77a682fe73d1f03c3649d5920ee822e9cf0 WatchSource:0}: Error finding container f67803051cfc98fcb874b56cf6ffc77a682fe73d1f03c3649d5920ee822e9cf0: Status 404 returned error can't find the container with id f67803051cfc98fcb874b56cf6ffc77a682fe73d1f03c3649d5920ee822e9cf0 Oct 31 13:57:08.871398 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:08.871363 5532 generic.go:356] "Generic (PLEG): container finished" podID="56ae5236-c47d-4182-9d85-9b8131f11783" containerID="4d6f6adb0d400823dccc7f38bec3d4cd6355e91199507ddd479ce1775cb9abd1" exitCode=0 Oct 31 13:57:08.871813 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:08.871447 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" event={"ID":"56ae5236-c47d-4182-9d85-9b8131f11783","Type":"ContainerDied","Data":"4d6f6adb0d400823dccc7f38bec3d4cd6355e91199507ddd479ce1775cb9abd1"} Oct 31 13:57:08.871813 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:08.871536 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" event={"ID":"56ae5236-c47d-4182-9d85-9b8131f11783","Type":"ContainerStarted","Data":"f67803051cfc98fcb874b56cf6ffc77a682fe73d1f03c3649d5920ee822e9cf0"} Oct 31 13:57:10.082563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.082537 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" Oct 31 13:57:10.148539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.148453 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ae5236-c47d-4182-9d85-9b8131f11783-kube-api-access-bmq9x\" (UniqueName: \"kubernetes.io/projected/56ae5236-c47d-4182-9d85-9b8131f11783-kube-api-access-bmq9x\") pod \"56ae5236-c47d-4182-9d85-9b8131f11783\" (UID: \"56ae5236-c47d-4182-9d85-9b8131f11783\") " Oct 31 13:57:10.148720 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.148564 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-openshift-ca\") pod \"56ae5236-c47d-4182-9d85-9b8131f11783\" (UID: \"56ae5236-c47d-4182-9d85-9b8131f11783\") " Oct 31 13:57:10.148773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.148729 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-xks-ca\") pod \"56ae5236-c47d-4182-9d85-9b8131f11783\" (UID: \"56ae5236-c47d-4182-9d85-9b8131f11783\") " Oct 31 13:57:10.148990 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.148959 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-openshift-ca" pod "56ae5236-c47d-4182-9d85-9b8131f11783" (UID: "56ae5236-c47d-4182-9d85-9b8131f11783"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:10.149412 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.149392 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:10.150841 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.150814 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ae5236-c47d-4182-9d85-9b8131f11783-kube-api-access-bmq9x" pod "56ae5236-c47d-4182-9d85-9b8131f11783" (UID: "56ae5236-c47d-4182-9d85-9b8131f11783"). InnerVolumeSpecName "kube-api-access-bmq9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:10.151399 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.151370 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-xks-ca" pod "56ae5236-c47d-4182-9d85-9b8131f11783" (UID: "56ae5236-c47d-4182-9d85-9b8131f11783"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:57:10.232186 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.232150 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29"] Oct 31 13:57:10.251384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.251226 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmq9x\" (UniqueName: \"kubernetes.io/projected/56ae5236-c47d-4182-9d85-9b8131f11783-kube-api-access-bmq9x\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:10.251384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.251263 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/56ae5236-c47d-4182-9d85-9b8131f11783-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:10.906807 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.906771 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" event={"ID":"56ae5236-c47d-4182-9d85-9b8131f11783","Type":"ContainerDied","Data":"f67803051cfc98fcb874b56cf6ffc77a682fe73d1f03c3649d5920ee822e9cf0"} Oct 31 13:57:10.906807 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.906799 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd" Oct 31 13:57:10.907029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:10.906820 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f67803051cfc98fcb874b56cf6ffc77a682fe73d1f03c3649d5920ee822e9cf0" Oct 31 13:57:15.232575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.232537 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" podUID="82327512-b6dc-45ab-8d1c-3453cfa8bd15" containerName="payload-pre-processing" containerID="cri-o://ad85c6467af98caf4948f8f1ce7f7bb488a50e8b95df12057a89eebc9823a8fc" gracePeriod=25 Oct 31 13:57:15.562998 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.562967 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" Oct 31 13:57:15.646843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.646811 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82327512-b6dc-45ab-8d1c-3453cfa8bd15-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82327512-b6dc-45ab-8d1c-3453cfa8bd15-plugins-config-volume\") pod \"82327512-b6dc-45ab-8d1c-3453cfa8bd15\" (UID: \"82327512-b6dc-45ab-8d1c-3453cfa8bd15\") " Oct 31 13:57:15.647047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.646926 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82327512-b6dc-45ab-8d1c-3453cfa8bd15-kube-api-access-lj7h6\" (UniqueName: \"kubernetes.io/projected/82327512-b6dc-45ab-8d1c-3453cfa8bd15-kube-api-access-lj7h6\") pod \"82327512-b6dc-45ab-8d1c-3453cfa8bd15\" (UID: \"82327512-b6dc-45ab-8d1c-3453cfa8bd15\") " Oct 31 13:57:15.648345 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.648317 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82327512-b6dc-45ab-8d1c-3453cfa8bd15-plugins-config-volume" pod "82327512-b6dc-45ab-8d1c-3453cfa8bd15" (UID: "82327512-b6dc-45ab-8d1c-3453cfa8bd15"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:15.654184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.654154 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82327512-b6dc-45ab-8d1c-3453cfa8bd15-kube-api-access-lj7h6" pod "82327512-b6dc-45ab-8d1c-3453cfa8bd15" (UID: "82327512-b6dc-45ab-8d1c-3453cfa8bd15"). InnerVolumeSpecName "kube-api-access-lj7h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:15.749385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.749350 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82327512-b6dc-45ab-8d1c-3453cfa8bd15-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:15.749385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.749378 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj7h6\" (UniqueName: \"kubernetes.io/projected/82327512-b6dc-45ab-8d1c-3453cfa8bd15-kube-api-access-lj7h6\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:15.951935 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.951903 5532 generic.go:356] "Generic (PLEG): container finished" podID="82327512-b6dc-45ab-8d1c-3453cfa8bd15" containerID="ad85c6467af98caf4948f8f1ce7f7bb488a50e8b95df12057a89eebc9823a8fc" exitCode=0 Oct 31 13:57:15.952119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.951953 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" Oct 31 13:57:15.952119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.951967 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" event={"ID":"82327512-b6dc-45ab-8d1c-3453cfa8bd15","Type":"ContainerDied","Data":"ad85c6467af98caf4948f8f1ce7f7bb488a50e8b95df12057a89eebc9823a8fc"} Oct 31 13:57:15.952119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.952034 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29" event={"ID":"82327512-b6dc-45ab-8d1c-3453cfa8bd15","Type":"ContainerDied","Data":"1f4a097575384b5cec5af8a119e100cf10346217e9dd3f8dea6320ef861a22ad"} Oct 31 13:57:15.952119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.952055 5532 scope.go:122] "RemoveContainer" containerID="ad85c6467af98caf4948f8f1ce7f7bb488a50e8b95df12057a89eebc9823a8fc" Oct 31 13:57:15.963994 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.963973 5532 scope.go:122] "RemoveContainer" containerID="ad85c6467af98caf4948f8f1ce7f7bb488a50e8b95df12057a89eebc9823a8fc" Oct 31 13:57:15.964282 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:57:15.964264 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad85c6467af98caf4948f8f1ce7f7bb488a50e8b95df12057a89eebc9823a8fc\": container with ID starting with ad85c6467af98caf4948f8f1ce7f7bb488a50e8b95df12057a89eebc9823a8fc not found: ID does not exist" containerID="ad85c6467af98caf4948f8f1ce7f7bb488a50e8b95df12057a89eebc9823a8fc" Oct 31 13:57:15.964336 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.964290 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad85c6467af98caf4948f8f1ce7f7bb488a50e8b95df12057a89eebc9823a8fc"} err="failed to get container status \"ad85c6467af98caf4948f8f1ce7f7bb488a50e8b95df12057a89eebc9823a8fc\": rpc error: code = NotFound desc = could not find container \"ad85c6467af98caf4948f8f1ce7f7bb488a50e8b95df12057a89eebc9823a8fc\": container with ID starting with ad85c6467af98caf4948f8f1ce7f7bb488a50e8b95df12057a89eebc9823a8fc not found: ID does not exist" Oct 31 13:57:15.979925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.979892 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29"] Oct 31 13:57:15.983984 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:15.983963 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6978849c67-ppk29"] Oct 31 13:57:16.006788 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:16.006763 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82327512-b6dc-45ab-8d1c-3453cfa8bd15" path="/var/lib/kubelet/pods/82327512-b6dc-45ab-8d1c-3453cfa8bd15/volumes" Oct 31 13:57:16.913064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:16.913027 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99"] Oct 31 13:57:16.913576 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:16.913284 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" podUID="da4caf91-ff54-45c0-98a1-7bbb51abb628" containerName="maas-api" containerID="cri-o://a198217546876a0e2160fde142b252e7f1c25cedb6256bb2dec705095c7bcd69" gracePeriod=30 Oct 31 13:57:17.018573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.018539 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx"] Oct 31 13:57:17.028650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.020132 5532 cpu_manager.go:414] "containerMap: removing container" podUID="56ae5236-c47d-4182-9d85-9b8131f11783" containerName="revoke-keys" Oct 31 13:57:17.028650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.020158 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ae5236-c47d-4182-9d85-9b8131f11783" containerName="revoke-keys" Oct 31 13:57:17.028650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.020274 5532 cpu_manager.go:414] "containerMap: removing container" podUID="82327512-b6dc-45ab-8d1c-3453cfa8bd15" containerName="payload-pre-processing" Oct 31 13:57:17.028650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.020282 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82327512-b6dc-45ab-8d1c-3453cfa8bd15" containerName="payload-pre-processing" Oct 31 13:57:17.028650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.020335 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" Oct 31 13:57:17.028650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.020344 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4723d399-8244-4fd0-81f2-ce35a33441c8" containerName="istio-proxy" Oct 31 13:57:17.028650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.020885 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="82327512-b6dc-45ab-8d1c-3453cfa8bd15" containerName="payload-pre-processing" Oct 31 13:57:17.028650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.020924 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ae5236-c47d-4182-9d85-9b8131f11783" containerName="revoke-keys" Oct 31 13:57:17.028650 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.027129 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.049705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.049672 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx"] Oct 31 13:57:17.202674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.192297 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e752232-e0de-4540-b22d-cf1928b53e31-istiod-ca-cert\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.203068 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.203036 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e752232-e0de-4540-b22d-cf1928b53e31-istio-podinfo\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.203294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.203274 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-envoy\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.204749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.203467 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.207196 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.207173 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-credential-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.207648 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.207424 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-data\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.208077 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.208057 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s85s4\" (UniqueName: \"kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-kube-api-access-s85s4\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.208388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.208369 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-certs\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.208565 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.208549 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-istio-token\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.312383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.311661 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-certs\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.312790 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.312342 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-certs\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.313442 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.313414 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-istio-token\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.313801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.313778 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e752232-e0de-4540-b22d-cf1928b53e31-istiod-ca-cert\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.314057 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.314037 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e752232-e0de-4540-b22d-cf1928b53e31-istio-podinfo\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.314237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.314215 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-envoy\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.314388 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.314371 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.314548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.314532 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-credential-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.314752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.314733 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-data\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.314933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.314913 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s85s4\" (UniqueName: \"kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-kube-api-access-s85s4\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.317159 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.316867 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.317259 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.317198 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e752232-e0de-4540-b22d-cf1928b53e31-istiod-ca-cert\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.317530 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.317461 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-credential-socket\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.317655 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.317541 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-data\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.331432 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.331381 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e752232-e0de-4540-b22d-cf1928b53e31-istio-podinfo\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.333014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.332982 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s85s4\" (UniqueName: \"kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-kube-api-access-s85s4\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.333455 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.333434 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-istio-token\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.334555 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.334471 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-envoy\") pod \"e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.363422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.363400 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:17.365807 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.365786 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:57:17.519270 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.519193 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-maas-api-tls\") pod \"da4caf91-ff54-45c0-98a1-7bbb51abb628\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " Oct 31 13:57:17.519270 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.519232 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da4caf91-ff54-45c0-98a1-7bbb51abb628-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4caf91-ff54-45c0-98a1-7bbb51abb628-openshift-service-ca\") pod \"da4caf91-ff54-45c0-98a1-7bbb51abb628\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " Oct 31 13:57:17.519510 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.519294 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-metrics-certs\") pod \"da4caf91-ff54-45c0-98a1-7bbb51abb628\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " Oct 31 13:57:17.519510 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.519346 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da4caf91-ff54-45c0-98a1-7bbb51abb628-kube-api-access-z85wm\" (UniqueName: \"kubernetes.io/projected/da4caf91-ff54-45c0-98a1-7bbb51abb628-kube-api-access-z85wm\") pod \"da4caf91-ff54-45c0-98a1-7bbb51abb628\" (UID: \"da4caf91-ff54-45c0-98a1-7bbb51abb628\") " Oct 31 13:57:17.519815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.519780 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4caf91-ff54-45c0-98a1-7bbb51abb628-openshift-service-ca" pod "da4caf91-ff54-45c0-98a1-7bbb51abb628" (UID: "da4caf91-ff54-45c0-98a1-7bbb51abb628"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:17.520173 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.520150 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4caf91-ff54-45c0-98a1-7bbb51abb628-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:17.521577 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.521549 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-maas-api-tls" pod "da4caf91-ff54-45c0-98a1-7bbb51abb628" (UID: "da4caf91-ff54-45c0-98a1-7bbb51abb628"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:57:17.521724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.521605 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da4caf91-ff54-45c0-98a1-7bbb51abb628-kube-api-access-z85wm" pod "da4caf91-ff54-45c0-98a1-7bbb51abb628" (UID: "da4caf91-ff54-45c0-98a1-7bbb51abb628"). InnerVolumeSpecName "kube-api-access-z85wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:17.521829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.521814 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-metrics-certs" pod "da4caf91-ff54-45c0-98a1-7bbb51abb628" (UID: "da4caf91-ff54-45c0-98a1-7bbb51abb628"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:57:17.552640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.551714 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7"] Oct 31 13:57:17.556892 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.556860 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-b06b08-m8hm7"] Oct 31 13:57:17.622054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.622020 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:17.622054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.622052 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z85wm\" (UniqueName: \"kubernetes.io/projected/da4caf91-ff54-45c0-98a1-7bbb51abb628-kube-api-access-z85wm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:17.622343 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.622064 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da4caf91-ff54-45c0-98a1-7bbb51abb628-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:17.669558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.669520 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx"] Oct 31 13:57:17.669793 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:57:17.669754 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e752232_e0de_4540_b22d_cf1928b53e31.slice/crio-66b8ff9e376733db93baceed6f142109a8424cf661f98a5309fd0cbb22bee6aa WatchSource:0}: Error finding container 66b8ff9e376733db93baceed6f142109a8424cf661f98a5309fd0cbb22bee6aa: Status 404 returned error can't find the container with id 66b8ff9e376733db93baceed6f142109a8424cf661f98a5309fd0cbb22bee6aa Oct 31 13:57:17.673557 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.673517 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:17.674942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.673775 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:17.674942 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.673828 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:17.789678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.789605 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f"] Oct 31 13:57:17.790475 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.790455 5532 cpu_manager.go:414] "containerMap: removing container" podUID="da4caf91-ff54-45c0-98a1-7bbb51abb628" containerName="maas-api" Oct 31 13:57:17.790525 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.790479 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da4caf91-ff54-45c0-98a1-7bbb51abb628" containerName="maas-api" Oct 31 13:57:17.790759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.790744 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="da4caf91-ff54-45c0-98a1-7bbb51abb628" containerName="maas-api" Oct 31 13:57:17.795405 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.795383 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" Oct 31 13:57:17.803361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.803313 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:57:17.803361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.803303 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:57:17.806305 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.806278 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f"] Oct 31 13:57:17.825877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.825842 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c68ba1d5-fc58f\" (UID: \"60c7b96f-35de-4027-b62d-8bfff3670de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" Oct 31 13:57:17.826026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.825930 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngwwb\" (UniqueName: \"kubernetes.io/projected/60c7b96f-35de-4027-b62d-8bfff3670de5-kube-api-access-ngwwb\") pod \"maas-api-revoke-sub-c68ba1d5-fc58f\" (UID: \"60c7b96f-35de-4027-b62d-8bfff3670de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" Oct 31 13:57:17.826154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.826136 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c68ba1d5-fc58f\" (UID: \"60c7b96f-35de-4027-b62d-8bfff3670de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" Oct 31 13:57:17.928069 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.928038 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c68ba1d5-fc58f\" (UID: \"60c7b96f-35de-4027-b62d-8bfff3670de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" Oct 31 13:57:17.928536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.928139 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c68ba1d5-fc58f\" (UID: \"60c7b96f-35de-4027-b62d-8bfff3670de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" Oct 31 13:57:17.928536 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.928188 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngwwb\" (UniqueName: \"kubernetes.io/projected/60c7b96f-35de-4027-b62d-8bfff3670de5-kube-api-access-ngwwb\") pod \"maas-api-revoke-sub-c68ba1d5-fc58f\" (UID: \"60c7b96f-35de-4027-b62d-8bfff3670de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" Oct 31 13:57:17.928900 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.928877 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c68ba1d5-fc58f\" (UID: \"60c7b96f-35de-4027-b62d-8bfff3670de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" Oct 31 13:57:17.930483 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.930460 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c68ba1d5-fc58f\" (UID: \"60c7b96f-35de-4027-b62d-8bfff3670de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" Oct 31 13:57:17.942486 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.942451 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngwwb\" (UniqueName: \"kubernetes.io/projected/60c7b96f-35de-4027-b62d-8bfff3670de5-kube-api-access-ngwwb\") pod \"maas-api-revoke-sub-c68ba1d5-fc58f\" (UID: \"60c7b96f-35de-4027-b62d-8bfff3670de5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" Oct 31 13:57:17.981928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.981891 5532 generic.go:356] "Generic (PLEG): container finished" podID="da4caf91-ff54-45c0-98a1-7bbb51abb628" containerID="a198217546876a0e2160fde142b252e7f1c25cedb6256bb2dec705095c7bcd69" exitCode=0 Oct 31 13:57:17.982123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.981943 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" Oct 31 13:57:17.982123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.981965 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" event={"ID":"da4caf91-ff54-45c0-98a1-7bbb51abb628","Type":"ContainerDied","Data":"a198217546876a0e2160fde142b252e7f1c25cedb6256bb2dec705095c7bcd69"} Oct 31 13:57:17.982123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.982029 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99" event={"ID":"da4caf91-ff54-45c0-98a1-7bbb51abb628","Type":"ContainerDied","Data":"8a8273d74f4fbc1afb78bd3b17a3442880086f06793d36be95f4036b0904708d"} Oct 31 13:57:17.982123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.982056 5532 scope.go:122] "RemoveContainer" containerID="a198217546876a0e2160fde142b252e7f1c25cedb6256bb2dec705095c7bcd69" Oct 31 13:57:17.984184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.984133 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" event={"ID":"5e752232-e0de-4540-b22d-cf1928b53e31","Type":"ContainerStarted","Data":"18900f8fe6db6eddc14133d74c5871d51ff5c4f5cd675a7b617a2a47c5cc0d3f"} Oct 31 13:57:17.984319 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.984202 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" event={"ID":"5e752232-e0de-4540-b22d-cf1928b53e31","Type":"ContainerStarted","Data":"66b8ff9e376733db93baceed6f142109a8424cf661f98a5309fd0cbb22bee6aa"} Oct 31 13:57:17.995220 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.995197 5532 scope.go:122] "RemoveContainer" containerID="a198217546876a0e2160fde142b252e7f1c25cedb6256bb2dec705095c7bcd69" Oct 31 13:57:17.995526 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:57:17.995507 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a198217546876a0e2160fde142b252e7f1c25cedb6256bb2dec705095c7bcd69\": container with ID starting with a198217546876a0e2160fde142b252e7f1c25cedb6256bb2dec705095c7bcd69 not found: ID does not exist" containerID="a198217546876a0e2160fde142b252e7f1c25cedb6256bb2dec705095c7bcd69" Oct 31 13:57:17.995628 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:17.995530 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a198217546876a0e2160fde142b252e7f1c25cedb6256bb2dec705095c7bcd69"} err="failed to get container status \"a198217546876a0e2160fde142b252e7f1c25cedb6256bb2dec705095c7bcd69\": rpc error: code = NotFound desc = could not find container \"a198217546876a0e2160fde142b252e7f1c25cedb6256bb2dec705095c7bcd69\": container with ID starting with a198217546876a0e2160fde142b252e7f1c25cedb6256bb2dec705095c7bcd69 not found: ID does not exist" Oct 31 13:57:18.004731 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:18.004708 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b224c68a-a55e-4485-a218-1fbdb9ab0437" path="/var/lib/kubelet/pods/b224c68a-a55e-4485-a218-1fbdb9ab0437/volumes" Oct 31 13:57:18.015464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:18.015422 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" podStartSLOduration=2.015383932 podStartE2EDuration="2.015383932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:18.010570425 +0000 UTC m=+3206.156314254" watchObservedRunningTime="2026-10-31 13:57:18.015383932 +0000 UTC m=+3206.161127760" Oct 31 13:57:18.036351 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:18.036326 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99"] Oct 31 13:57:18.040983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:18.040926 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-b06b08-775854bb77-d7n99"] Oct 31 13:57:18.117622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:18.117563 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" Oct 31 13:57:18.364104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:18.364021 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:18.365409 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:18.365382 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:57:18.365539 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:18.365438 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" podUID="5e752232-e0de-4540-b22d-cf1928b53e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:57:18.417160 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:57:18.417128 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c7b96f_35de_4027_b62d_8bfff3670de5.slice/crio-3e76b94f55c20ab923b6f3b92ad8109f506b631461ff8ef48a43ff41373f64ac WatchSource:0}: Error finding container 3e76b94f55c20ab923b6f3b92ad8109f506b631461ff8ef48a43ff41373f64ac: Status 404 returned error can't find the container with id 3e76b94f55c20ab923b6f3b92ad8109f506b631461ff8ef48a43ff41373f64ac Oct 31 13:57:18.417527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:18.417357 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f"] Oct 31 13:57:18.996204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:18.996107 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" event={"ID":"60c7b96f-35de-4027-b62d-8bfff3670de5","Type":"ContainerStarted","Data":"c6f3511d55390824e160a44129147d519db72cf5bdb4fca7ba89431606bb26d3"} Oct 31 13:57:18.996204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:18.996159 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" event={"ID":"60c7b96f-35de-4027-b62d-8bfff3670de5","Type":"ContainerStarted","Data":"3e76b94f55c20ab923b6f3b92ad8109f506b631461ff8ef48a43ff41373f64ac"} Oct 31 13:57:19.023895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:19.023838 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" podStartSLOduration=2.023819652 podStartE2EDuration="2.023819652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:19.01587329 +0000 UTC m=+3207.161617120" watchObservedRunningTime="2026-10-31 13:57:19.023819652 +0000 UTC m=+3207.169563481" Oct 31 13:57:19.364041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:19.363946 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:57:19.364041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:19.363998 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" podUID="5e752232-e0de-4540-b22d-cf1928b53e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:57:20.006839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:20.006799 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da4caf91-ff54-45c0-98a1-7bbb51abb628" path="/var/lib/kubelet/pods/da4caf91-ff54-45c0-98a1-7bbb51abb628/volumes" Oct 31 13:57:20.007361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:20.007339 5532 generic.go:356] "Generic (PLEG): container finished" podID="60c7b96f-35de-4027-b62d-8bfff3670de5" containerID="c6f3511d55390824e160a44129147d519db72cf5bdb4fca7ba89431606bb26d3" exitCode=0 Oct 31 13:57:20.007446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:20.007421 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" event={"ID":"60c7b96f-35de-4027-b62d-8bfff3670de5","Type":"ContainerDied","Data":"c6f3511d55390824e160a44129147d519db72cf5bdb4fca7ba89431606bb26d3"} Oct 31 13:57:20.364074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:20.364036 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:57:20.364273 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:20.364103 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" podUID="5e752232-e0de-4540-b22d-cf1928b53e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:57:21.275031 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.274962 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" Oct 31 13:57:21.306537 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.306495 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c7b96f-35de-4027-b62d-8bfff3670de5-kube-api-access-ngwwb\" (UniqueName: \"kubernetes.io/projected/60c7b96f-35de-4027-b62d-8bfff3670de5-kube-api-access-ngwwb\") pod \"60c7b96f-35de-4027-b62d-8bfff3670de5\" (UID: \"60c7b96f-35de-4027-b62d-8bfff3670de5\") " Oct 31 13:57:21.307147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.307118 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-openshift-ca\") pod \"60c7b96f-35de-4027-b62d-8bfff3670de5\" (UID: \"60c7b96f-35de-4027-b62d-8bfff3670de5\") " Oct 31 13:57:21.307267 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.307216 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-xks-ca\") pod \"60c7b96f-35de-4027-b62d-8bfff3670de5\" (UID: \"60c7b96f-35de-4027-b62d-8bfff3670de5\") " Oct 31 13:57:21.308573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.308542 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-openshift-ca" pod "60c7b96f-35de-4027-b62d-8bfff3670de5" (UID: "60c7b96f-35de-4027-b62d-8bfff3670de5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:21.309126 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.309099 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c7b96f-35de-4027-b62d-8bfff3670de5-kube-api-access-ngwwb" pod "60c7b96f-35de-4027-b62d-8bfff3670de5" (UID: "60c7b96f-35de-4027-b62d-8bfff3670de5"). InnerVolumeSpecName "kube-api-access-ngwwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:21.310337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.310308 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-xks-ca" pod "60c7b96f-35de-4027-b62d-8bfff3670de5" (UID: "60c7b96f-35de-4027-b62d-8bfff3670de5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:57:21.370122 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.370073 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:57:21.370345 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.370145 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" podUID="5e752232-e0de-4540-b22d-cf1928b53e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:57:21.409674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.409643 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngwwb\" (UniqueName: \"kubernetes.io/projected/60c7b96f-35de-4027-b62d-8bfff3670de5-kube-api-access-ngwwb\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:21.409674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.409671 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:21.409883 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.409684 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60c7b96f-35de-4027-b62d-8bfff3670de5-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:21.570555 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.570460 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6"] Oct 31 13:57:21.571267 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.571250 5532 cpu_manager.go:414] "containerMap: removing container" podUID="60c7b96f-35de-4027-b62d-8bfff3670de5" containerName="revoke-keys" Oct 31 13:57:21.571331 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.571268 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c7b96f-35de-4027-b62d-8bfff3670de5" containerName="revoke-keys" Oct 31 13:57:21.571450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.571435 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c7b96f-35de-4027-b62d-8bfff3670de5" containerName="revoke-keys" Oct 31 13:57:21.576227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.576201 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.579347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.579318 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6"] Oct 31 13:57:21.581911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.581885 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-abdcb1-openshift-default-dockercfg-jnc5d\"" Oct 31 13:57:21.612860 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.612820 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-envoy\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.613039 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.612952 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-data\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.613141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.613118 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-certs\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.613192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.613160 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f40b74-b0b4-4f94-a08e-3bd06c339993-istiod-ca-cert\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.613298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.613271 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.613415 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.613388 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-podinfo\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.613495 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.613479 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpfnz\" (UniqueName: \"kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-kube-api-access-cpfnz\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.613541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.613519 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-credential-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.613617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.613590 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-token\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.714891 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.714859 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-podinfo\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.715065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.714936 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpfnz\" (UniqueName: \"kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-kube-api-access-cpfnz\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.715065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.714991 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-credential-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.715065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.715045 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-token\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.715222 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.715080 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-envoy\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.715222 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.715170 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-data\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.715365 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.715344 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-certs\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.715443 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.715395 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f40b74-b0b4-4f94-a08e-3bd06c339993-istiod-ca-cert\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.715509 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.715488 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.715573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.715518 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-credential-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.715668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.715606 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-data\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.715725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.715672 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-certs\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.715895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.715871 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-socket\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.716186 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.716164 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f40b74-b0b4-4f94-a08e-3bd06c339993-istiod-ca-cert\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.717449 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.717423 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-envoy\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.717652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.717633 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-podinfo\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.728703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.728677 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-token\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.729019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.728998 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpfnz\" (UniqueName: \"kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-kube-api-access-cpfnz\") pod \"e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:21.896066 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:21.896032 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:22.102434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.101196 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" event={"ID":"60c7b96f-35de-4027-b62d-8bfff3670de5","Type":"ContainerDied","Data":"3e76b94f55c20ab923b6f3b92ad8109f506b631461ff8ef48a43ff41373f64ac"} Oct 31 13:57:22.102434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.101257 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e76b94f55c20ab923b6f3b92ad8109f506b631461ff8ef48a43ff41373f64ac" Oct 31 13:57:22.102434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.101326 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f" Oct 31 13:57:22.192380 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.192304 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk"] Oct 31 13:57:22.197442 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.197419 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.204150 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.204124 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-a39e7e-openshift-default-dockercfg-x278n\"" Oct 31 13:57:22.209916 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.209888 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk"] Oct 31 13:57:22.223694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.223668 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6"] Oct 31 13:57:22.225864 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:57:22.225732 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32f40b74_b0b4_4f94_a08e_3bd06c339993.slice/crio-b0c59aa7e0a29b426dd91f5700f8e569bbe8de89fc1bc867452435520d99f78e WatchSource:0}: Error finding container b0c59aa7e0a29b426dd91f5700f8e569bbe8de89fc1bc867452435520d99f78e: Status 404 returned error can't find the container with id b0c59aa7e0a29b426dd91f5700f8e569bbe8de89fc1bc867452435520d99f78e Oct 31 13:57:22.229257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.229226 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-data\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.229419 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.229397 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-credential-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.229530 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.229470 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.229620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.229546 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istiod-ca-cert\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.229698 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.229619 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-podinfo\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.229760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.229696 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75b85\" (UniqueName: \"kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-kube-api-access-75b85\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.229836 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.229779 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-envoy\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.229836 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.229806 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-token\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.229955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.229876 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-certs\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.236868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.236834 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:22.236975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.236902 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:22.236975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.236930 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:22.331033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.330998 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-data\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.331422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.331116 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-credential-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.331422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.331183 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.331422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.331345 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istiod-ca-cert\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.331422 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.331390 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-podinfo\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.331682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.331425 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-data\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.331682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.331444 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-credential-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.331682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.331511 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75b85\" (UniqueName: \"kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-kube-api-access-75b85\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.331682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.331516 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-socket\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.331682 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.331666 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-envoy\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.331925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.331703 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-token\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.331925 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.331780 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-certs\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.332084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.332063 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istiod-ca-cert\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.332125 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.332094 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-certs\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.333848 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.333828 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-envoy\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.334147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.334128 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-podinfo\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.371699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.371128 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-token\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.371699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.371274 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:57:22.371699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.371319 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" podUID="5e752232-e0de-4540-b22d-cf1928b53e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:57:22.371699 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.371330 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75b85\" (UniqueName: \"kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-kube-api-access-75b85\") pod \"e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.513916 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.513815 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:22.850672 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.850178 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk"] Oct 31 13:57:22.892439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.891433 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:22.892439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.891536 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:22.892439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:22.891587 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:23.122677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:23.122542 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" event={"ID":"32f40b74-b0b4-4f94-a08e-3bd06c339993","Type":"ContainerStarted","Data":"29213cd50f64676d77addf2ee1e589b1106e6d1846f5c85a285be0d6bbd6b2bd"} Oct 31 13:57:23.122677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:23.122672 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" event={"ID":"32f40b74-b0b4-4f94-a08e-3bd06c339993","Type":"ContainerStarted","Data":"b0c59aa7e0a29b426dd91f5700f8e569bbe8de89fc1bc867452435520d99f78e"} Oct 31 13:57:23.124796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:23.124767 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" event={"ID":"b7361fc2-13c5-47db-ae28-cf18c0acfef6","Type":"ContainerStarted","Data":"06ba0522c955faef7a073a5f299d9dfc4ce110f43f5fd3cd2d479c87f5d6fe9b"} Oct 31 13:57:23.124953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:23.124812 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" event={"ID":"b7361fc2-13c5-47db-ae28-cf18c0acfef6","Type":"ContainerStarted","Data":"ba353cd63d51e06e9a513788640f7f1c29615039f225735f4c041829d39616e3"} Oct 31 13:57:23.197618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:23.197232 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" podStartSLOduration=2.197207613 podStartE2EDuration="2.197207613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:23.181982044 +0000 UTC m=+3211.327725894" watchObservedRunningTime="2026-10-31 13:57:23.197207613 +0000 UTC m=+3211.342951442" Oct 31 13:57:23.214526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:23.212016 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" podStartSLOduration=1.211997162 podStartE2EDuration="1.211997162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:23.206761813 +0000 UTC m=+3211.352505648" watchObservedRunningTime="2026-10-31 13:57:23.211997162 +0000 UTC m=+3211.357740991" Oct 31 13:57:23.372145 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:23.372107 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:57:23.372801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:23.372731 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" podUID="5e752232-e0de-4540-b22d-cf1928b53e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:57:23.514939 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:23.514897 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:23.903282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:23.903244 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:23.909375 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:23.909339 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 31 13:57:23.909684 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:23.909655 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" podUID="32f40b74-b0b4-4f94-a08e-3bd06c339993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 31 13:57:24.365564 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:24.365018 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" start-of-body= Oct 31 13:57:24.365564 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:24.365082 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" podUID="5e752232-e0de-4540-b22d-cf1928b53e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.27:15021/healthz/ready\": dial tcp 10.132.1.27:15021: connect: connection refused" Oct 31 13:57:24.516325 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:24.516286 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:57:24.516772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:24.516341 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" podUID="b7361fc2-13c5-47db-ae28-cf18c0acfef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 31 13:57:24.904816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:24.896533 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 31 13:57:24.904816 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:24.904320 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" podUID="32f40b74-b0b4-4f94-a08e-3bd06c339993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 31 13:57:25.420349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:25.395973 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:25.420349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:25.397803 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:25.420349 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:25.416827 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 13:57:25.515695 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:25.515334 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:57:25.515695 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:25.515392 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" podUID="b7361fc2-13c5-47db-ae28-cf18c0acfef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 31 13:57:25.522828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:25.521353 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw"] Oct 31 13:57:25.522828 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:25.521582 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerName="istio-proxy" containerID="cri-o://26f047daca6017c7c02df7147ef527c347d7161d4239fcf08f8e23df8b8f9036" gracePeriod=30 Oct 31 13:57:25.909527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:25.902094 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 31 13:57:25.909527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:25.902159 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" podUID="32f40b74-b0b4-4f94-a08e-3bd06c339993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 31 13:57:26.299958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.299843 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs"] Oct 31 13:57:26.323701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.323666 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs"] Oct 31 13:57:26.323909 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.323796 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" Oct 31 13:57:26.391615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.391121 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:57:26.391615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.391294 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:57:26.425108 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.425071 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67824477-81be-4ff1-82e4-62c65528526a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs\" (UID: \"67824477-81be-4ff1-82e4-62c65528526a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" Oct 31 13:57:26.425427 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.425400 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5mkx\" (UniqueName: \"kubernetes.io/projected/67824477-81be-4ff1-82e4-62c65528526a-kube-api-access-b5mkx\") pod \"maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs\" (UID: \"67824477-81be-4ff1-82e4-62c65528526a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" Oct 31 13:57:26.425526 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.425508 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67824477-81be-4ff1-82e4-62c65528526a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs\" (UID: \"67824477-81be-4ff1-82e4-62c65528526a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" Oct 31 13:57:26.515321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.515147 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:57:26.515321 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.515228 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" podUID="b7361fc2-13c5-47db-ae28-cf18c0acfef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 31 13:57:26.527425 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.527393 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5mkx\" (UniqueName: \"kubernetes.io/projected/67824477-81be-4ff1-82e4-62c65528526a-kube-api-access-b5mkx\") pod \"maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs\" (UID: \"67824477-81be-4ff1-82e4-62c65528526a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" Oct 31 13:57:26.527832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.527446 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67824477-81be-4ff1-82e4-62c65528526a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs\" (UID: \"67824477-81be-4ff1-82e4-62c65528526a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" Oct 31 13:57:26.527832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.527572 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67824477-81be-4ff1-82e4-62c65528526a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs\" (UID: \"67824477-81be-4ff1-82e4-62c65528526a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" Oct 31 13:57:26.528521 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.528495 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67824477-81be-4ff1-82e4-62c65528526a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs\" (UID: \"67824477-81be-4ff1-82e4-62c65528526a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" Oct 31 13:57:26.530817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.530792 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67824477-81be-4ff1-82e4-62c65528526a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs\" (UID: \"67824477-81be-4ff1-82e4-62c65528526a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" Oct 31 13:57:26.561662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.560751 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5mkx\" (UniqueName: \"kubernetes.io/projected/67824477-81be-4ff1-82e4-62c65528526a-kube-api-access-b5mkx\") pod \"maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs\" (UID: \"67824477-81be-4ff1-82e4-62c65528526a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" Oct 31 13:57:26.704965 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.704930 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" Oct 31 13:57:26.897459 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.897085 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 31 13:57:26.897459 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:26.897159 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" podUID="32f40b74-b0b4-4f94-a08e-3bd06c339993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 31 13:57:27.092889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:27.092852 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs"] Oct 31 13:57:27.094177 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:57:27.094150 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67824477_81be_4ff1_82e4_62c65528526a.slice/crio-b0959f9bdc145eb65f9d6e20035401e6727d59984af6e12b1e8bb99597032653 WatchSource:0}: Error finding container b0959f9bdc145eb65f9d6e20035401e6727d59984af6e12b1e8bb99597032653: Status 404 returned error can't find the container with id b0959f9bdc145eb65f9d6e20035401e6727d59984af6e12b1e8bb99597032653 Oct 31 13:57:27.220571 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:27.220531 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" event={"ID":"67824477-81be-4ff1-82e4-62c65528526a","Type":"ContainerStarted","Data":"b0959f9bdc145eb65f9d6e20035401e6727d59984af6e12b1e8bb99597032653"} Oct 31 13:57:27.514634 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:27.514548 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:57:27.514829 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:27.514631 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" podUID="b7361fc2-13c5-47db-ae28-cf18c0acfef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 31 13:57:27.898634 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:27.898460 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 31 13:57:27.898634 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:27.898522 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" podUID="32f40b74-b0b4-4f94-a08e-3bd06c339993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 31 13:57:28.230027 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.229991 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6"] Oct 31 13:57:28.234946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.234921 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" event={"ID":"67824477-81be-4ff1-82e4-62c65528526a","Type":"ContainerStarted","Data":"07cb4f9b9c4f4a08a429d1005334cab2c59d062ab01cf01c7a32844e60350ea8"} Oct 31 13:57:28.235097 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.235005 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" Oct 31 13:57:28.266754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.266722 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e50b22\"" Oct 31 13:57:28.267879 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.267854 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e50b22-dockercfg-87m5r\"" Oct 31 13:57:28.309889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.308629 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" podStartSLOduration=2.308613034 podStartE2EDuration="2.308613034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:28.287183202 +0000 UTC m=+3216.432927032" watchObservedRunningTime="2026-10-31 13:57:28.308613034 +0000 UTC m=+3216.454356867" Oct 31 13:57:28.309889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.308962 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6"] Oct 31 13:57:28.317663 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.317631 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz"] Oct 31 13:57:28.324821 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.324793 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" Oct 31 13:57:28.330387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.330094 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz"] Oct 31 13:57:28.432677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.432624 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a200fc-140b-432d-8bf2-47f7a892bca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6\" (UID: \"d7a200fc-140b-432d-8bf2-47f7a892bca7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" Oct 31 13:57:28.433094 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.433071 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqzjm\" (UniqueName: \"kubernetes.io/projected/d7a200fc-140b-432d-8bf2-47f7a892bca7-kube-api-access-kqzjm\") pod \"payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6\" (UID: \"d7a200fc-140b-432d-8bf2-47f7a892bca7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" Oct 31 13:57:28.519458 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.516322 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:57:28.519458 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.516382 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" podUID="b7361fc2-13c5-47db-ae28-cf18c0acfef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 31 13:57:28.546615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.546391 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a200fc-140b-432d-8bf2-47f7a892bca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6\" (UID: \"d7a200fc-140b-432d-8bf2-47f7a892bca7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" Oct 31 13:57:28.546615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.546525 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7lhf\" (UniqueName: \"kubernetes.io/projected/93d476d7-a18f-4ac0-a11d-3fb38b57d429-kube-api-access-c7lhf\") pod \"payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz\" (UID: \"93d476d7-a18f-4ac0-a11d-3fb38b57d429\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" Oct 31 13:57:28.546887 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.546629 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqzjm\" (UniqueName: \"kubernetes.io/projected/d7a200fc-140b-432d-8bf2-47f7a892bca7-kube-api-access-kqzjm\") pod \"payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6\" (UID: \"d7a200fc-140b-432d-8bf2-47f7a892bca7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" Oct 31 13:57:28.546887 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.546778 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d476d7-a18f-4ac0-a11d-3fb38b57d429-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz\" (UID: \"93d476d7-a18f-4ac0-a11d-3fb38b57d429\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" Oct 31 13:57:28.567625 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.567030 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a200fc-140b-432d-8bf2-47f7a892bca7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6\" (UID: \"d7a200fc-140b-432d-8bf2-47f7a892bca7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" Oct 31 13:57:28.668618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.656299 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7lhf\" (UniqueName: \"kubernetes.io/projected/93d476d7-a18f-4ac0-a11d-3fb38b57d429-kube-api-access-c7lhf\") pod \"payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz\" (UID: \"93d476d7-a18f-4ac0-a11d-3fb38b57d429\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" Oct 31 13:57:28.668618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.656615 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d476d7-a18f-4ac0-a11d-3fb38b57d429-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz\" (UID: \"93d476d7-a18f-4ac0-a11d-3fb38b57d429\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" Oct 31 13:57:28.668618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.657711 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d476d7-a18f-4ac0-a11d-3fb38b57d429-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz\" (UID: \"93d476d7-a18f-4ac0-a11d-3fb38b57d429\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" Oct 31 13:57:28.668618 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.662778 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqzjm\" (UniqueName: \"kubernetes.io/projected/d7a200fc-140b-432d-8bf2-47f7a892bca7-kube-api-access-kqzjm\") pod \"payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6\" (UID: \"d7a200fc-140b-432d-8bf2-47f7a892bca7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" Oct 31 13:57:28.818930 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.798488 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7lhf\" (UniqueName: \"kubernetes.io/projected/93d476d7-a18f-4ac0-a11d-3fb38b57d429-kube-api-access-c7lhf\") pod \"payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz\" (UID: \"93d476d7-a18f-4ac0-a11d-3fb38b57d429\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" Oct 31 13:57:28.899627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.897563 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" start-of-body= Oct 31 13:57:28.899627 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.897664 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" podUID="32f40b74-b0b4-4f94-a08e-3bd06c339993" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.29:15021/healthz/ready\": dial tcp 10.132.1.29:15021: connect: connection refused" Oct 31 13:57:28.916054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:28.916023 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" Oct 31 13:57:29.003087 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:29.003055 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" Oct 31 13:57:29.284224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:29.283268 5532 generic.go:356] "Generic (PLEG): container finished" podID="67824477-81be-4ff1-82e4-62c65528526a" containerID="07cb4f9b9c4f4a08a429d1005334cab2c59d062ab01cf01c7a32844e60350ea8" exitCode=0 Oct 31 13:57:29.284224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:29.283398 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" event={"ID":"67824477-81be-4ff1-82e4-62c65528526a","Type":"ContainerDied","Data":"07cb4f9b9c4f4a08a429d1005334cab2c59d062ab01cf01c7a32844e60350ea8"} Oct 31 13:57:29.287330 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:29.287300 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6"] Oct 31 13:57:29.373174 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:29.373141 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz"] Oct 31 13:57:29.374489 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:57:29.374463 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d476d7_a18f_4ac0_a11d_3fb38b57d429.slice/crio-eeeb6550dc81233757e7a08cbe29fe128f56293793aa9fc06834c8c97c35c48b WatchSource:0}: Error finding container eeeb6550dc81233757e7a08cbe29fe128f56293793aa9fc06834c8c97c35c48b: Status 404 returned error can't find the container with id eeeb6550dc81233757e7a08cbe29fe128f56293793aa9fc06834c8c97c35c48b Oct 31 13:57:29.517825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:29.515994 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" start-of-body= Oct 31 13:57:29.517825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:29.516055 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" podUID="b7361fc2-13c5-47db-ae28-cf18c0acfef6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.30:15021/healthz/ready\": dial tcp 10.132.1.30:15021: connect: connection refused" Oct 31 13:57:29.904177 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:29.904145 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:29.904633 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:29.904615 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:29.905261 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:29.905233 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 13:57:30.297450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.297342 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" event={"ID":"93d476d7-a18f-4ac0-a11d-3fb38b57d429","Type":"ContainerStarted","Data":"7adb0c9dbacd4095c97cbcb0aa76e8fe122a0b844564d22decb0818ff546c2b6"} Oct 31 13:57:30.297450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.297418 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" Oct 31 13:57:30.297450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.297447 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" event={"ID":"93d476d7-a18f-4ac0-a11d-3fb38b57d429","Type":"ContainerStarted","Data":"eeeb6550dc81233757e7a08cbe29fe128f56293793aa9fc06834c8c97c35c48b"} Oct 31 13:57:30.299542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.299509 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" event={"ID":"d7a200fc-140b-432d-8bf2-47f7a892bca7","Type":"ContainerStarted","Data":"047ec799462ee77cef1d5e4eb6956933ff93e35b105d22853cbacfb9f746738f"} Oct 31 13:57:30.299702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.299562 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" event={"ID":"d7a200fc-140b-432d-8bf2-47f7a892bca7","Type":"ContainerStarted","Data":"1b254c2273cc3871edcc4280b93c45a5e24c65e96b9dd10f38ccb8a7c819bcad"} Oct 31 13:57:30.322582 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.322514 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" podStartSLOduration=2.322493124 podStartE2EDuration="2.322493124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:30.319707727 +0000 UTC m=+3218.465451552" watchObservedRunningTime="2026-10-31 13:57:30.322493124 +0000 UTC m=+3218.468236953" Oct 31 13:57:30.348257 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.348196 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" podStartSLOduration=2.348177295 podStartE2EDuration="2.348177295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:30.346848848 +0000 UTC m=+3218.492592676" watchObservedRunningTime="2026-10-31 13:57:30.348177295 +0000 UTC m=+3218.493921128" Oct 31 13:57:30.599187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.599142 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:30.600492 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.600468 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:30.601457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.601434 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 13:57:30.653784 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.653754 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" Oct 31 13:57:30.757513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.753938 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67824477-81be-4ff1-82e4-62c65528526a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67824477-81be-4ff1-82e4-62c65528526a-maas-api-xks-ca\") pod \"67824477-81be-4ff1-82e4-62c65528526a\" (UID: \"67824477-81be-4ff1-82e4-62c65528526a\") " Oct 31 13:57:30.757513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.754010 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67824477-81be-4ff1-82e4-62c65528526a-kube-api-access-b5mkx\" (UniqueName: \"kubernetes.io/projected/67824477-81be-4ff1-82e4-62c65528526a-kube-api-access-b5mkx\") pod \"67824477-81be-4ff1-82e4-62c65528526a\" (UID: \"67824477-81be-4ff1-82e4-62c65528526a\") " Oct 31 13:57:30.757513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.754157 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67824477-81be-4ff1-82e4-62c65528526a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67824477-81be-4ff1-82e4-62c65528526a-maas-api-openshift-ca\") pod \"67824477-81be-4ff1-82e4-62c65528526a\" (UID: \"67824477-81be-4ff1-82e4-62c65528526a\") " Oct 31 13:57:30.757513 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.755265 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67824477-81be-4ff1-82e4-62c65528526a-maas-api-openshift-ca" pod "67824477-81be-4ff1-82e4-62c65528526a" (UID: "67824477-81be-4ff1-82e4-62c65528526a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:30.757923 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.757870 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67824477-81be-4ff1-82e4-62c65528526a-maas-api-xks-ca" pod "67824477-81be-4ff1-82e4-62c65528526a" (UID: "67824477-81be-4ff1-82e4-62c65528526a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:57:30.759667 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.759368 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67824477-81be-4ff1-82e4-62c65528526a-kube-api-access-b5mkx" pod "67824477-81be-4ff1-82e4-62c65528526a" (UID: "67824477-81be-4ff1-82e4-62c65528526a"). InnerVolumeSpecName "kube-api-access-b5mkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:30.856491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.856390 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67824477-81be-4ff1-82e4-62c65528526a-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:30.856491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.856432 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5mkx\" (UniqueName: \"kubernetes.io/projected/67824477-81be-4ff1-82e4-62c65528526a-kube-api-access-b5mkx\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:30.856491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.856448 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67824477-81be-4ff1-82e4-62c65528526a-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:30.918209 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.918180 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:57:30.958121 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.958072 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-certs\") pod \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " Oct 31 13:57:30.958402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.958155 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-credential-socket\") pod \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " Oct 31 13:57:30.958402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.958232 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-envoy\") pod \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " Oct 31 13:57:30.958402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.958277 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-podinfo\") pod \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " Oct 31 13:57:30.958402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.958389 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-kube-api-access-4n8f8\" (UniqueName: \"kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-kube-api-access-4n8f8\") pod \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " Oct 31 13:57:30.958712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.958413 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-certs" pod "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" (UID: "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:57:30.958712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.958516 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-credential-socket" pod "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" (UID: "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:57:30.958712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.958531 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istiod-ca-cert\") pod \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " Oct 31 13:57:30.958712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.958582 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-token\" (UniqueName: \"kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-token\") pod \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " Oct 31 13:57:30.958712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.958668 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-socket\") pod \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " Oct 31 13:57:30.958959 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.958777 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-data\") pod \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\" (UID: \"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d\") " Oct 31 13:57:30.959009 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.958989 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istiod-ca-cert" pod "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" (UID: "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:30.959641 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.959570 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:30.959754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.959647 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:30.959754 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.959664 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:30.959940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.959915 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-socket" pod "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" (UID: "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:57:30.959998 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.959923 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-data" pod "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" (UID: "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:57:30.961547 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.961508 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-envoy" pod "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" (UID: "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:57:30.961887 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.961862 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-kube-api-access-4n8f8" pod "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" (UID: "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d"). InnerVolumeSpecName "kube-api-access-4n8f8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:30.962308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.962275 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-token" pod "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" (UID: "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:30.962471 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:30.962448 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-podinfo" pod "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" (UID: "d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:57:31.062105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.061837 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:31.062105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.061871 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:31.062105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.061889 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n8f8\" (UniqueName: \"kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-kube-api-access-4n8f8\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:31.062105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.061904 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:31.062105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.061919 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:31.062105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.061933 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:31.313310 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.313277 5532 generic.go:356] "Generic (PLEG): container finished" podID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerID="26f047daca6017c7c02df7147ef527c347d7161d4239fcf08f8e23df8b8f9036" exitCode=0 Oct 31 13:57:31.313477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.313355 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" Oct 31 13:57:31.313477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.313361 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" event={"ID":"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d","Type":"ContainerDied","Data":"26f047daca6017c7c02df7147ef527c347d7161d4239fcf08f8e23df8b8f9036"} Oct 31 13:57:31.313477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.313417 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw" event={"ID":"d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d","Type":"ContainerDied","Data":"fa8dfd059f174d7d4f32541d99c923de70c12fb9f8f0ec3c25187f69e059ae6c"} Oct 31 13:57:31.313477 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.313440 5532 scope.go:122] "RemoveContainer" containerID="26f047daca6017c7c02df7147ef527c347d7161d4239fcf08f8e23df8b8f9036" Oct 31 13:57:31.315491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.315470 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" Oct 31 13:57:31.315647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.315505 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs" event={"ID":"67824477-81be-4ff1-82e4-62c65528526a","Type":"ContainerDied","Data":"b0959f9bdc145eb65f9d6e20035401e6727d59984af6e12b1e8bb99597032653"} Oct 31 13:57:31.315647 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.315559 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0959f9bdc145eb65f9d6e20035401e6727d59984af6e12b1e8bb99597032653" Oct 31 13:57:31.315885 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.315835 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" Oct 31 13:57:31.329622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.327189 5532 scope.go:122] "RemoveContainer" containerID="965f8963eb8cb9c73c2014686e459ee16690946c6eed7444a93069b411636bdb" Oct 31 13:57:31.340337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.340308 5532 scope.go:122] "RemoveContainer" containerID="26f047daca6017c7c02df7147ef527c347d7161d4239fcf08f8e23df8b8f9036" Oct 31 13:57:31.340679 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:57:31.340654 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f047daca6017c7c02df7147ef527c347d7161d4239fcf08f8e23df8b8f9036\": container with ID starting with 26f047daca6017c7c02df7147ef527c347d7161d4239fcf08f8e23df8b8f9036 not found: ID does not exist" containerID="26f047daca6017c7c02df7147ef527c347d7161d4239fcf08f8e23df8b8f9036" Oct 31 13:57:31.340764 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.340703 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f047daca6017c7c02df7147ef527c347d7161d4239fcf08f8e23df8b8f9036"} err="failed to get container status \"26f047daca6017c7c02df7147ef527c347d7161d4239fcf08f8e23df8b8f9036\": rpc error: code = NotFound desc = could not find container \"26f047daca6017c7c02df7147ef527c347d7161d4239fcf08f8e23df8b8f9036\": container with ID starting with 26f047daca6017c7c02df7147ef527c347d7161d4239fcf08f8e23df8b8f9036 not found: ID does not exist" Oct 31 13:57:31.340764 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.340726 5532 scope.go:122] "RemoveContainer" containerID="965f8963eb8cb9c73c2014686e459ee16690946c6eed7444a93069b411636bdb" Oct 31 13:57:31.341014 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:57:31.340996 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"965f8963eb8cb9c73c2014686e459ee16690946c6eed7444a93069b411636bdb\": container with ID starting with 965f8963eb8cb9c73c2014686e459ee16690946c6eed7444a93069b411636bdb not found: ID does not exist" containerID="965f8963eb8cb9c73c2014686e459ee16690946c6eed7444a93069b411636bdb" Oct 31 13:57:31.341072 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.341018 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"965f8963eb8cb9c73c2014686e459ee16690946c6eed7444a93069b411636bdb"} err="failed to get container status \"965f8963eb8cb9c73c2014686e459ee16690946c6eed7444a93069b411636bdb\": rpc error: code = NotFound desc = could not find container \"965f8963eb8cb9c73c2014686e459ee16690946c6eed7444a93069b411636bdb\": container with ID starting with 965f8963eb8cb9c73c2014686e459ee16690946c6eed7444a93069b411636bdb not found: ID does not exist" Oct 31 13:57:31.349839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.349812 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw"] Oct 31 13:57:31.354394 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.354369 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-b06b08-openshift-default-5cd658c6b5-srzrw"] Oct 31 13:57:31.441904 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.441871 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r"] Oct 31 13:57:31.442685 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.442668 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerName="istio-proxy" Oct 31 13:57:31.442759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.442686 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerName="istio-proxy" Oct 31 13:57:31.442759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.442711 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerName="istio-proxy" Oct 31 13:57:31.442759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.442716 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerName="istio-proxy" Oct 31 13:57:31.442759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.442730 5532 cpu_manager.go:414] "containerMap: removing container" podUID="67824477-81be-4ff1-82e4-62c65528526a" containerName="revoke-keys" Oct 31 13:57:31.442759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.442735 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67824477-81be-4ff1-82e4-62c65528526a" containerName="revoke-keys" Oct 31 13:57:31.442916 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.442865 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerName="istio-proxy" Oct 31 13:57:31.442916 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.442877 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="67824477-81be-4ff1-82e4-62c65528526a" containerName="revoke-keys" Oct 31 13:57:31.448484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.448466 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.454356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.453589 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-33c10a-openshift-default-dockercfg-ptdwg\"" Oct 31 13:57:31.458916 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.458824 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r"] Oct 31 13:57:31.467817 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.467789 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-data\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.467956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.467920 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-istio-token\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.468048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.467970 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-envoy\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.468141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.468122 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-certs\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.468208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.468175 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e8516d-253e-454e-b928-c24938cfe79b-istio-podinfo\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.468263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.468241 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-socket\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.468320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.468277 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-credential-socket\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.468320 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.468295 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-758dw\" (UniqueName: \"kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-kube-api-access-758dw\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.468442 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.468414 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e8516d-253e-454e-b928-c24938cfe79b-istiod-ca-cert\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.570109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.570026 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-istio-token\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.570109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.570067 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-envoy\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.570322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.570180 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-certs\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.570322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.570234 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e8516d-253e-454e-b928-c24938cfe79b-istio-podinfo\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.570322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.570287 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-socket\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.570322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.570316 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-credential-socket\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.570544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.570337 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-758dw\" (UniqueName: \"kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-kube-api-access-758dw\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.570544 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.570406 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e8516d-253e-454e-b928-c24938cfe79b-istiod-ca-cert\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.570724 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.570537 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-data\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.570823 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.570795 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-socket\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.570887 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.570830 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-credential-socket\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.570987 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.570965 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-data\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.571123 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.571073 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-certs\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.571362 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.571344 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e8516d-253e-454e-b928-c24938cfe79b-istiod-ca-cert\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.572657 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.572636 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-envoy\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.572772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.572756 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e8516d-253e-454e-b928-c24938cfe79b-istio-podinfo\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.583668 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.583639 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-istio-token\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.583808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.583767 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-758dw\" (UniqueName: \"kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-kube-api-access-758dw\") pod \"e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:31.764188 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:57:31.764146 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:57:31.766480 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:31.766459 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:32.006509 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:32.006476 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" path="/var/lib/kubelet/pods/d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d/volumes" Oct 31 13:57:32.075573 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:32.075515 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r"] Oct 31 13:57:32.075939 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:57:32.075915 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53e8516d_253e_454e_b928_c24938cfe79b.slice/crio-448705a2ae6113ffacca5f271f323d94f3083ebb62790a912702e3ce40e58421 WatchSource:0}: Error finding container 448705a2ae6113ffacca5f271f323d94f3083ebb62790a912702e3ce40e58421: Status 404 returned error can't find the container with id 448705a2ae6113ffacca5f271f323d94f3083ebb62790a912702e3ce40e58421 Oct 31 13:57:32.078989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:32.078957 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:32.079089 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:32.079023 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:32.079089 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:32.079049 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:32.327999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:32.327905 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" event={"ID":"53e8516d-253e-454e-b928-c24938cfe79b","Type":"ContainerStarted","Data":"bc44cb99387738a539634d9154a08d357b665467d3fcd59a713bc5379bcc9551"} Oct 31 13:57:32.327999 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:32.327972 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" event={"ID":"53e8516d-253e-454e-b928-c24938cfe79b","Type":"ContainerStarted","Data":"448705a2ae6113ffacca5f271f323d94f3083ebb62790a912702e3ce40e58421"} Oct 31 13:57:32.355611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:32.355532 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" podStartSLOduration=1.355514015 podStartE2EDuration="1.355514015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:32.353446455 +0000 UTC m=+3220.499190285" watchObservedRunningTime="2026-10-31 13:57:32.355514015 +0000 UTC m=+3220.501257847" Oct 31 13:57:32.767619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:32.766678 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:33.684620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:33.684345 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw"] Oct 31 13:57:33.685434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:33.685212 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" podUID="b4b2679c-6948-4251-aec3-cdd30eb68497" containerName="payload-processing" containerID="cri-o://b951b0c7bc811951f85162a57d431d6350dde05372b03d0f332ce7fab26698f0" gracePeriod=30 Oct 31 13:57:33.693383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:33.691657 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz"] Oct 31 13:57:33.693383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:33.691912 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" podUID="f6a13f9a-dafa-4839-ac03-247b821dd5c7" containerName="payload-pre-processing" containerID="cri-o://323b94ff4cfd11c0d526402f90e94431fe6ba873b89612c5aea0867fe2333482" gracePeriod=30 Oct 31 13:57:33.785446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:33.785412 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Oct 31 13:57:33.785549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:33.785468 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" podUID="53e8516d-253e-454e-b928-c24938cfe79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Oct 31 13:57:34.304642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.304358 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb"] Oct 31 13:57:34.309433 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.309408 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-dafbce-mzbwb"] Oct 31 13:57:34.394732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.388745 5532 generic.go:356] "Generic (PLEG): container finished" podID="b4b2679c-6948-4251-aec3-cdd30eb68497" containerID="b951b0c7bc811951f85162a57d431d6350dde05372b03d0f332ce7fab26698f0" exitCode=0 Oct 31 13:57:34.394732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.388814 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" event={"ID":"b4b2679c-6948-4251-aec3-cdd30eb68497","Type":"ContainerDied","Data":"b951b0c7bc811951f85162a57d431d6350dde05372b03d0f332ce7fab26698f0"} Oct 31 13:57:34.394732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.390644 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" Oct 31 13:57:34.394732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.390660 5532 generic.go:356] "Generic (PLEG): container finished" podID="f6a13f9a-dafa-4839-ac03-247b821dd5c7" containerID="323b94ff4cfd11c0d526402f90e94431fe6ba873b89612c5aea0867fe2333482" exitCode=0 Oct 31 13:57:34.394732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.390850 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" event={"ID":"f6a13f9a-dafa-4839-ac03-247b821dd5c7","Type":"ContainerDied","Data":"323b94ff4cfd11c0d526402f90e94431fe6ba873b89612c5aea0867fe2333482"} Oct 31 13:57:34.394732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.390892 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" event={"ID":"f6a13f9a-dafa-4839-ac03-247b821dd5c7","Type":"ContainerDied","Data":"de7b8ab404f6ce700ca8828717ca79777851ed3423bae6048a66986c3e7d0f1e"} Oct 31 13:57:34.394732 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.390915 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de7b8ab404f6ce700ca8828717ca79777851ed3423bae6048a66986c3e7d0f1e" Oct 31 13:57:34.415053 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.415022 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6a13f9a-dafa-4839-ac03-247b821dd5c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a13f9a-dafa-4839-ac03-247b821dd5c7-plugins-config-volume\") pod \"f6a13f9a-dafa-4839-ac03-247b821dd5c7\" (UID: \"f6a13f9a-dafa-4839-ac03-247b821dd5c7\") " Oct 31 13:57:34.415184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.415097 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6a13f9a-dafa-4839-ac03-247b821dd5c7-kube-api-access-55jm2\" (UniqueName: \"kubernetes.io/projected/f6a13f9a-dafa-4839-ac03-247b821dd5c7-kube-api-access-55jm2\") pod \"f6a13f9a-dafa-4839-ac03-247b821dd5c7\" (UID: \"f6a13f9a-dafa-4839-ac03-247b821dd5c7\") " Oct 31 13:57:34.416002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.415971 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6a13f9a-dafa-4839-ac03-247b821dd5c7-plugins-config-volume" pod "f6a13f9a-dafa-4839-ac03-247b821dd5c7" (UID: "f6a13f9a-dafa-4839-ac03-247b821dd5c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:34.418071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.418042 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6a13f9a-dafa-4839-ac03-247b821dd5c7-kube-api-access-55jm2" pod "f6a13f9a-dafa-4839-ac03-247b821dd5c7" (UID: "f6a13f9a-dafa-4839-ac03-247b821dd5c7"). InnerVolumeSpecName "kube-api-access-55jm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:34.484169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.484136 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" Oct 31 13:57:34.516807 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.516776 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4b2679c-6948-4251-aec3-cdd30eb68497-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b2679c-6948-4251-aec3-cdd30eb68497-plugins-config-volume\") pod \"b4b2679c-6948-4251-aec3-cdd30eb68497\" (UID: \"b4b2679c-6948-4251-aec3-cdd30eb68497\") " Oct 31 13:57:34.517033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.517016 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4b2679c-6948-4251-aec3-cdd30eb68497-kube-api-access-wqr9h\" (UniqueName: \"kubernetes.io/projected/b4b2679c-6948-4251-aec3-cdd30eb68497-kube-api-access-wqr9h\") pod \"b4b2679c-6948-4251-aec3-cdd30eb68497\" (UID: \"b4b2679c-6948-4251-aec3-cdd30eb68497\") " Oct 31 13:57:34.517132 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.517116 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4b2679c-6948-4251-aec3-cdd30eb68497-plugins-config-volume" pod "b4b2679c-6948-4251-aec3-cdd30eb68497" (UID: "b4b2679c-6948-4251-aec3-cdd30eb68497"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:34.517708 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.517687 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6a13f9a-dafa-4839-ac03-247b821dd5c7-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:34.517819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.517709 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55jm2\" (UniqueName: \"kubernetes.io/projected/f6a13f9a-dafa-4839-ac03-247b821dd5c7-kube-api-access-55jm2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:34.517819 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.517723 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4b2679c-6948-4251-aec3-cdd30eb68497-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:34.519531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.519501 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4b2679c-6948-4251-aec3-cdd30eb68497-kube-api-access-wqr9h" pod "b4b2679c-6948-4251-aec3-cdd30eb68497" (UID: "b4b2679c-6948-4251-aec3-cdd30eb68497"). InnerVolumeSpecName "kube-api-access-wqr9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:34.619872 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.619791 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqr9h\" (UniqueName: \"kubernetes.io/projected/b4b2679c-6948-4251-aec3-cdd30eb68497-kube-api-access-wqr9h\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:34.769393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.769357 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Oct 31 13:57:34.770081 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:34.770048 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" podUID="53e8516d-253e-454e-b928-c24938cfe79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Oct 31 13:57:35.402113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:35.402081 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" Oct 31 13:57:35.402113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:35.402106 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz" Oct 31 13:57:35.402113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:35.402107 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw" event={"ID":"b4b2679c-6948-4251-aec3-cdd30eb68497","Type":"ContainerDied","Data":"007fe33f7911dd3eaf23119a3649e741971208d02a45f8ff526d89bb1a5bb863"} Oct 31 13:57:35.402523 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:35.402180 5532 scope.go:122] "RemoveContainer" containerID="b951b0c7bc811951f85162a57d431d6350dde05372b03d0f332ce7fab26698f0" Oct 31 13:57:35.434017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:35.433990 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz"] Oct 31 13:57:35.462441 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:35.455041 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-dafbce-754f9454f5-n7gdz"] Oct 31 13:57:35.479673 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:35.479583 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw"] Oct 31 13:57:35.488158 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:35.484102 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-dafbce-f6dc47458-nbmcw"] Oct 31 13:57:35.767916 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:35.767809 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Oct 31 13:57:35.767916 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:35.767875 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" podUID="53e8516d-253e-454e-b928-c24938cfe79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Oct 31 13:57:36.006764 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.006728 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a83d7f4a-1ad3-4721-aaa6-69a7408d5697" path="/var/lib/kubelet/pods/a83d7f4a-1ad3-4721-aaa6-69a7408d5697/volumes" Oct 31 13:57:36.007231 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.007214 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4b2679c-6948-4251-aec3-cdd30eb68497" path="/var/lib/kubelet/pods/b4b2679c-6948-4251-aec3-cdd30eb68497/volumes" Oct 31 13:57:36.007637 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.007623 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6a13f9a-dafa-4839-ac03-247b821dd5c7" path="/var/lib/kubelet/pods/f6a13f9a-dafa-4839-ac03-247b821dd5c7/volumes" Oct 31 13:57:36.491395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.491355 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c"] Oct 31 13:57:36.492830 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.492804 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b4b2679c-6948-4251-aec3-cdd30eb68497" containerName="payload-processing" Oct 31 13:57:36.492978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.492963 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4b2679c-6948-4251-aec3-cdd30eb68497" containerName="payload-processing" Oct 31 13:57:36.493172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.493156 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f6a13f9a-dafa-4839-ac03-247b821dd5c7" containerName="payload-pre-processing" Oct 31 13:57:36.493276 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.493258 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6a13f9a-dafa-4839-ac03-247b821dd5c7" containerName="payload-pre-processing" Oct 31 13:57:36.493494 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.493477 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6a13f9a-dafa-4839-ac03-247b821dd5c7" containerName="payload-pre-processing" Oct 31 13:57:36.493579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.493498 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b8b3d6-33ce-4ddf-a4ae-64a1d573c49d" containerName="istio-proxy" Oct 31 13:57:36.493579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.493511 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4b2679c-6948-4251-aec3-cdd30eb68497" containerName="payload-processing" Oct 31 13:57:36.501906 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.501887 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.510189 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.509953 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c"] Oct 31 13:57:36.579502 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.579399 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-certs\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.579725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.579556 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-credential-socket\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.579725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.579560 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc"] Oct 31 13:57:36.579725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.579588 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667e09dd-9d9e-431a-8720-c860ef93e504-istiod-ca-cert\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.579887 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.579725 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-data\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.579887 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.579859 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvfb7\" (UniqueName: \"kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-kube-api-access-rvfb7\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.580019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.580000 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-envoy\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.580389 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.580117 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-socket\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.580389 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.580199 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-istio-token\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.580389 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.580248 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667e09dd-9d9e-431a-8720-c860ef93e504-istio-podinfo\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.586894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.586865 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-9f9103-openshift-default-dockercfg-8xml8\"" Oct 31 13:57:36.683998 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.683961 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-certs\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.684195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.684039 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-credential-socket\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.684195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.684071 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667e09dd-9d9e-431a-8720-c860ef93e504-istiod-ca-cert\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.684195 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.684142 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-data\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.684384 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.684272 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvfb7\" (UniqueName: \"kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-kube-api-access-rvfb7\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.684441 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.684420 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-envoy\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.684501 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.684466 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-socket\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.685331 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.685085 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-data\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.685331 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.685281 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-credential-socket\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.685533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.685441 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-socket\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.685616 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.685575 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667e09dd-9d9e-431a-8720-c860ef93e504-istiod-ca-cert\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.685761 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.685738 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-istio-token\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.685834 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.685817 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667e09dd-9d9e-431a-8720-c860ef93e504-istio-podinfo\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.686136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.686111 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-certs\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.688268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.688237 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-envoy\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.689052 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.689028 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667e09dd-9d9e-431a-8720-c860ef93e504-istio-podinfo\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.700467 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.700283 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvfb7\" (UniqueName: \"kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-kube-api-access-rvfb7\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.701326 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.701300 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-istio-token\") pod \"e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:36.766960 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.766881 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Oct 31 13:57:36.767109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.766956 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" podUID="53e8516d-253e-454e-b928-c24938cfe79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Oct 31 13:57:36.827868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:36.827827 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:37.469548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:37.469516 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c"] Oct 31 13:57:37.469889 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:57:37.469629 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod667e09dd_9d9e_431a_8720_c860ef93e504.slice/crio-831abc3019db0ec021607ec172c3e302cf13a519347a2721f988831015656d32 WatchSource:0}: Error finding container 831abc3019db0ec021607ec172c3e302cf13a519347a2721f988831015656d32: Status 404 returned error can't find the container with id 831abc3019db0ec021607ec172c3e302cf13a519347a2721f988831015656d32 Oct 31 13:57:37.472773 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:37.472743 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:37.472851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:37.472810 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:37.472851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:37.472840 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:57:37.767504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:37.767399 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" start-of-body= Oct 31 13:57:37.767504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:37.767463 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" podUID="53e8516d-253e-454e-b928-c24938cfe79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.34:15021/healthz/ready\": dial tcp 10.132.1.34:15021: connect: connection refused" Oct 31 13:57:38.430347 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:38.430307 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" event={"ID":"667e09dd-9d9e-431a-8720-c860ef93e504","Type":"ContainerStarted","Data":"83fd679e0d8219e8032d6806dedad415d23adaf87642a15d780678cd8cd4a694"} Oct 31 13:57:38.430540 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:38.430369 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" event={"ID":"667e09dd-9d9e-431a-8720-c860ef93e504","Type":"ContainerStarted","Data":"831abc3019db0ec021607ec172c3e302cf13a519347a2721f988831015656d32"} Oct 31 13:57:38.471156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:38.471090 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" podStartSLOduration=2.471069621 podStartE2EDuration="2.471069621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:38.462963514 +0000 UTC m=+3226.608707340" watchObservedRunningTime="2026-10-31 13:57:38.471069621 +0000 UTC m=+3226.616813452" Oct 31 13:57:38.828119 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:38.828029 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:38.831782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:38.831752 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:38.832715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:38.832691 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:38.843022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:38.834101 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 13:57:38.843022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:38.835883 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:39.439165 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:39.439133 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:39.440039 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:39.440017 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:57:41.317607 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.317563 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" Oct 31 13:57:41.580747 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.580625 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" podUID="6343c05e-f79e-4832-9073-cf8e4e97a1f4" containerName="payload-processing" containerID="cri-o://697275ebcb2d8fef4176736195aec58b64c7e19711ed484c3fbe43c8c21e7dad" gracePeriod=25 Oct 31 13:57:41.642242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.642209 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd"] Oct 31 13:57:41.647094 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.647070 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8"] Oct 31 13:57:41.647243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.647208 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" Oct 31 13:57:41.652303 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.652282 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" Oct 31 13:57:41.656128 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.656104 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd"] Oct 31 13:57:41.662170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.662144 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8"] Oct 31 13:57:41.709970 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.709922 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:57:41.710098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.710017 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:57:41.799243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.799209 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-251c725c-qhhpd\" (UID: \"459a9c12-801e-456c-9da6-f64f297c9459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" Oct 31 13:57:41.799428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.799276 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d9d1c3-kpfp8\" (UID: \"719ab474-6de3-473c-8dd3-6240703d5de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" Oct 31 13:57:41.799428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.799324 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d9d1c3-kpfp8\" (UID: \"719ab474-6de3-473c-8dd3-6240703d5de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" Oct 31 13:57:41.799428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.799350 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-251c725c-qhhpd\" (UID: \"459a9c12-801e-456c-9da6-f64f297c9459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" Oct 31 13:57:41.799428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.799373 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lbz8\" (UniqueName: \"kubernetes.io/projected/719ab474-6de3-473c-8dd3-6240703d5de8-kube-api-access-7lbz8\") pod \"maas-api-revoke-sub-d8d9d1c3-kpfp8\" (UID: \"719ab474-6de3-473c-8dd3-6240703d5de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" Oct 31 13:57:41.799703 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.799551 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47t7l\" (UniqueName: \"kubernetes.io/projected/459a9c12-801e-456c-9da6-f64f297c9459-kube-api-access-47t7l\") pod \"maas-api-revoke-sub-251c725c-qhhpd\" (UID: \"459a9c12-801e-456c-9da6-f64f297c9459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" Oct 31 13:57:41.878079 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.878056 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" Oct 31 13:57:41.907858 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.907822 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-251c725c-qhhpd\" (UID: \"459a9c12-801e-456c-9da6-f64f297c9459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" Oct 31 13:57:41.908047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.907920 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d9d1c3-kpfp8\" (UID: \"719ab474-6de3-473c-8dd3-6240703d5de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" Oct 31 13:57:41.908047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.907991 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d9d1c3-kpfp8\" (UID: \"719ab474-6de3-473c-8dd3-6240703d5de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" Oct 31 13:57:41.908047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.908032 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-251c725c-qhhpd\" (UID: \"459a9c12-801e-456c-9da6-f64f297c9459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" Oct 31 13:57:41.908236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.908068 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lbz8\" (UniqueName: \"kubernetes.io/projected/719ab474-6de3-473c-8dd3-6240703d5de8-kube-api-access-7lbz8\") pod \"maas-api-revoke-sub-d8d9d1c3-kpfp8\" (UID: \"719ab474-6de3-473c-8dd3-6240703d5de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" Oct 31 13:57:41.908356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.908338 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47t7l\" (UniqueName: \"kubernetes.io/projected/459a9c12-801e-456c-9da6-f64f297c9459-kube-api-access-47t7l\") pod \"maas-api-revoke-sub-251c725c-qhhpd\" (UID: \"459a9c12-801e-456c-9da6-f64f297c9459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" Oct 31 13:57:41.910086 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.910013 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8d9d1c3-kpfp8\" (UID: \"719ab474-6de3-473c-8dd3-6240703d5de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" Oct 31 13:57:41.910644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.910580 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-251c725c-qhhpd\" (UID: \"459a9c12-801e-456c-9da6-f64f297c9459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" Oct 31 13:57:41.912694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.912671 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-xks-ca\") pod \"maas-api-revoke-sub-251c725c-qhhpd\" (UID: \"459a9c12-801e-456c-9da6-f64f297c9459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" Oct 31 13:57:41.914968 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.914944 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8d9d1c3-kpfp8\" (UID: \"719ab474-6de3-473c-8dd3-6240703d5de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" Oct 31 13:57:41.923023 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.922234 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47t7l\" (UniqueName: \"kubernetes.io/projected/459a9c12-801e-456c-9da6-f64f297c9459-kube-api-access-47t7l\") pod \"maas-api-revoke-sub-251c725c-qhhpd\" (UID: \"459a9c12-801e-456c-9da6-f64f297c9459\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" Oct 31 13:57:41.946851 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.941094 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lbz8\" (UniqueName: \"kubernetes.io/projected/719ab474-6de3-473c-8dd3-6240703d5de8-kube-api-access-7lbz8\") pod \"maas-api-revoke-sub-d8d9d1c3-kpfp8\" (UID: \"719ab474-6de3-473c-8dd3-6240703d5de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" Oct 31 13:57:41.975680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.975639 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" Oct 31 13:57:41.987861 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:41.985101 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" Oct 31 13:57:42.009163 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.009131 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6343c05e-f79e-4832-9073-cf8e4e97a1f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6343c05e-f79e-4832-9073-cf8e4e97a1f4-plugins-config-volume\") pod \"6343c05e-f79e-4832-9073-cf8e4e97a1f4\" (UID: \"6343c05e-f79e-4832-9073-cf8e4e97a1f4\") " Oct 31 13:57:42.009395 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.009372 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6343c05e-f79e-4832-9073-cf8e4e97a1f4-kube-api-access-tr5ch\" (UniqueName: \"kubernetes.io/projected/6343c05e-f79e-4832-9073-cf8e4e97a1f4-kube-api-access-tr5ch\") pod \"6343c05e-f79e-4832-9073-cf8e4e97a1f4\" (UID: \"6343c05e-f79e-4832-9073-cf8e4e97a1f4\") " Oct 31 13:57:42.010777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.010748 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6343c05e-f79e-4832-9073-cf8e4e97a1f4-plugins-config-volume" pod "6343c05e-f79e-4832-9073-cf8e4e97a1f4" (UID: "6343c05e-f79e-4832-9073-cf8e4e97a1f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:42.022391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.022348 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6343c05e-f79e-4832-9073-cf8e4e97a1f4-kube-api-access-tr5ch" pod "6343c05e-f79e-4832-9073-cf8e4e97a1f4" (UID: "6343c05e-f79e-4832-9073-cf8e4e97a1f4"). InnerVolumeSpecName "kube-api-access-tr5ch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:42.121033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.118363 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6343c05e-f79e-4832-9073-cf8e4e97a1f4-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:42.121033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.118402 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr5ch\" (UniqueName: \"kubernetes.io/projected/6343c05e-f79e-4832-9073-cf8e4e97a1f4-kube-api-access-tr5ch\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:42.431701 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.420777 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" Oct 31 13:57:42.446137 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.439908 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd"] Oct 31 13:57:42.487830 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.487794 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8"] Oct 31 13:57:42.494252 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.494226 5532 generic.go:356] "Generic (PLEG): container finished" podID="6343c05e-f79e-4832-9073-cf8e4e97a1f4" containerID="697275ebcb2d8fef4176736195aec58b64c7e19711ed484c3fbe43c8c21e7dad" exitCode=0 Oct 31 13:57:42.498649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.494434 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" event={"ID":"6343c05e-f79e-4832-9073-cf8e4e97a1f4","Type":"ContainerDied","Data":"697275ebcb2d8fef4176736195aec58b64c7e19711ed484c3fbe43c8c21e7dad"} Oct 31 13:57:42.498649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.494607 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" event={"ID":"6343c05e-f79e-4832-9073-cf8e4e97a1f4","Type":"ContainerDied","Data":"34249cb3f89824363b6194a0fa6f9b37565fda29c631b205d2c15ae613e34550"} Oct 31 13:57:42.498649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.494643 5532 scope.go:122] "RemoveContainer" containerID="697275ebcb2d8fef4176736195aec58b64c7e19711ed484c3fbe43c8c21e7dad" Oct 31 13:57:42.498649 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.494464 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc" Oct 31 13:57:42.511438 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.511130 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" event={"ID":"459a9c12-801e-456c-9da6-f64f297c9459","Type":"ContainerStarted","Data":"490d992f99ac768c639594e6bdeced27182478e128d67c66127fd954eff6335c"} Oct 31 13:57:42.512661 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.512617 5532 scope.go:122] "RemoveContainer" containerID="697275ebcb2d8fef4176736195aec58b64c7e19711ed484c3fbe43c8c21e7dad" Oct 31 13:57:42.512951 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:57:42.512921 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697275ebcb2d8fef4176736195aec58b64c7e19711ed484c3fbe43c8c21e7dad\": container with ID starting with 697275ebcb2d8fef4176736195aec58b64c7e19711ed484c3fbe43c8c21e7dad not found: ID does not exist" containerID="697275ebcb2d8fef4176736195aec58b64c7e19711ed484c3fbe43c8c21e7dad" Oct 31 13:57:42.513035 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.512961 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697275ebcb2d8fef4176736195aec58b64c7e19711ed484c3fbe43c8c21e7dad"} err="failed to get container status \"697275ebcb2d8fef4176736195aec58b64c7e19711ed484c3fbe43c8c21e7dad\": rpc error: code = NotFound desc = could not find container \"697275ebcb2d8fef4176736195aec58b64c7e19711ed484c3fbe43c8c21e7dad\": container with ID starting with 697275ebcb2d8fef4176736195aec58b64c7e19711ed484c3fbe43c8c21e7dad not found: ID does not exist" Oct 31 13:57:42.552385 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.552353 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc"] Oct 31 13:57:42.557104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:42.557078 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5286ea-5f5f55bd9c-hmkdc"] Oct 31 13:57:43.594164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.593934 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" event={"ID":"719ab474-6de3-473c-8dd3-6240703d5de8","Type":"ContainerStarted","Data":"6ca16a30f17d4c3f27a084f08bea64fed2547eb9ef5c90dc75c03d527703e672"} Oct 31 13:57:43.594164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.594002 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" event={"ID":"719ab474-6de3-473c-8dd3-6240703d5de8","Type":"ContainerStarted","Data":"9ab8c325916c8b88c80c5de8afd40989cef0d16c4df308fb47eaf4e8c9a68313"} Oct 31 13:57:43.601029 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.600801 5532 generic.go:356] "Generic (PLEG): container finished" podID="459a9c12-801e-456c-9da6-f64f297c9459" containerID="2e6b0baaf11ca6922d735ec4ac3df64dfa17e77122cce0c0ecd380ea2a2103ec" exitCode=0 Oct 31 13:57:43.601211 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.601069 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" event={"ID":"459a9c12-801e-456c-9da6-f64f297c9459","Type":"ContainerDied","Data":"2e6b0baaf11ca6922d735ec4ac3df64dfa17e77122cce0c0ecd380ea2a2103ec"} Oct 31 13:57:43.623726 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.623675 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" podStartSLOduration=2.623659877 podStartE2EDuration="2.623659877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:57:43.61751576 +0000 UTC m=+3231.763259588" watchObservedRunningTime="2026-10-31 13:57:43.623659877 +0000 UTC m=+3231.769403706" Oct 31 13:57:43.841853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.841773 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd"] Oct 31 13:57:43.842619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.842589 5532 cpu_manager.go:414] "containerMap: removing container" podUID="6343c05e-f79e-4832-9073-cf8e4e97a1f4" containerName="payload-processing" Oct 31 13:57:43.842679 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.842621 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6343c05e-f79e-4832-9073-cf8e4e97a1f4" containerName="payload-processing" Oct 31 13:57:43.842768 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.842759 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="6343c05e-f79e-4832-9073-cf8e4e97a1f4" containerName="payload-processing" Oct 31 13:57:43.846716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.846698 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" Oct 31 13:57:43.856013 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.855932 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd"] Oct 31 13:57:43.974707 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.974673 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd\" (UID: \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" Oct 31 13:57:43.974889 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.974720 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sscw8\" (UniqueName: \"kubernetes.io/projected/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-kube-api-access-sscw8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd\" (UID: \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" Oct 31 13:57:43.974939 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:43.974915 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd\" (UID: \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" Oct 31 13:57:44.007073 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.007037 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6343c05e-f79e-4832-9073-cf8e4e97a1f4" path="/var/lib/kubelet/pods/6343c05e-f79e-4832-9073-cf8e4e97a1f4/volumes" Oct 31 13:57:44.085619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.081587 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd\" (UID: \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" Oct 31 13:57:44.085619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.081840 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd\" (UID: \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" Oct 31 13:57:44.085619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.081890 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sscw8\" (UniqueName: \"kubernetes.io/projected/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-kube-api-access-sscw8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd\" (UID: \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" Oct 31 13:57:44.085619 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.083123 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd\" (UID: \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" Oct 31 13:57:44.096638 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.096520 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd\" (UID: \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" Oct 31 13:57:44.101313 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.101235 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sscw8\" (UniqueName: \"kubernetes.io/projected/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-kube-api-access-sscw8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd\" (UID: \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" Oct 31 13:57:44.163895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.163863 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" Oct 31 13:57:44.455055 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:57:44.455025 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f0f0fb5_fac0_4d72_aab0_3c77dd3182c0.slice/crio-f069862cce5a48a534abf08b529fd570aabb305f729a9a310f0f4dfe7010cdb4 WatchSource:0}: Error finding container f069862cce5a48a534abf08b529fd570aabb305f729a9a310f0f4dfe7010cdb4: Status 404 returned error can't find the container with id f069862cce5a48a534abf08b529fd570aabb305f729a9a310f0f4dfe7010cdb4 Oct 31 13:57:44.455300 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.455274 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd"] Oct 31 13:57:44.611725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.611689 5532 generic.go:356] "Generic (PLEG): container finished" podID="719ab474-6de3-473c-8dd3-6240703d5de8" containerID="6ca16a30f17d4c3f27a084f08bea64fed2547eb9ef5c90dc75c03d527703e672" exitCode=0 Oct 31 13:57:44.612193 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.611780 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" event={"ID":"719ab474-6de3-473c-8dd3-6240703d5de8","Type":"ContainerDied","Data":"6ca16a30f17d4c3f27a084f08bea64fed2547eb9ef5c90dc75c03d527703e672"} Oct 31 13:57:44.613522 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.613496 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" event={"ID":"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0","Type":"ContainerStarted","Data":"f069862cce5a48a534abf08b529fd570aabb305f729a9a310f0f4dfe7010cdb4"} Oct 31 13:57:44.811904 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.811877 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" Oct 31 13:57:44.997041 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.996960 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/459a9c12-801e-456c-9da6-f64f297c9459-kube-api-access-47t7l\" (UniqueName: \"kubernetes.io/projected/459a9c12-801e-456c-9da6-f64f297c9459-kube-api-access-47t7l\") pod \"459a9c12-801e-456c-9da6-f64f297c9459\" (UID: \"459a9c12-801e-456c-9da6-f64f297c9459\") " Oct 31 13:57:44.997204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.997074 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-openshift-ca\") pod \"459a9c12-801e-456c-9da6-f64f297c9459\" (UID: \"459a9c12-801e-456c-9da6-f64f297c9459\") " Oct 31 13:57:44.997204 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.997187 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-xks-ca\") pod \"459a9c12-801e-456c-9da6-f64f297c9459\" (UID: \"459a9c12-801e-456c-9da6-f64f297c9459\") " Oct 31 13:57:44.997522 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.997484 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-openshift-ca" pod "459a9c12-801e-456c-9da6-f64f297c9459" (UID: "459a9c12-801e-456c-9da6-f64f297c9459"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:44.997811 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.997795 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:44.999180 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.999153 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/459a9c12-801e-456c-9da6-f64f297c9459-kube-api-access-47t7l" pod "459a9c12-801e-456c-9da6-f64f297c9459" (UID: "459a9c12-801e-456c-9da6-f64f297c9459"). InnerVolumeSpecName "kube-api-access-47t7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:44.999287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:44.999180 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-xks-ca" pod "459a9c12-801e-456c-9da6-f64f297c9459" (UID: "459a9c12-801e-456c-9da6-f64f297c9459"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:57:45.099878 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:45.099845 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/459a9c12-801e-456c-9da6-f64f297c9459-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:45.099878 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:45.099882 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47t7l\" (UniqueName: \"kubernetes.io/projected/459a9c12-801e-456c-9da6-f64f297c9459-kube-api-access-47t7l\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:45.623752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:45.623724 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" Oct 31 13:57:45.623752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:45.623734 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd" event={"ID":"459a9c12-801e-456c-9da6-f64f297c9459","Type":"ContainerDied","Data":"490d992f99ac768c639594e6bdeced27182478e128d67c66127fd954eff6335c"} Oct 31 13:57:45.624308 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:45.623796 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="490d992f99ac768c639594e6bdeced27182478e128d67c66127fd954eff6335c" Oct 31 13:57:45.641437 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:45.641402 5532 generic.go:356] "Generic (PLEG): container finished" podID="8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0" containerID="c6f5213640633118891d437c61bd307e032ea7e9f82f91d9dc119b20e547fdf8" exitCode=0 Oct 31 13:57:45.641654 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:45.641568 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" event={"ID":"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0","Type":"ContainerDied","Data":"c6f5213640633118891d437c61bd307e032ea7e9f82f91d9dc119b20e547fdf8"} Oct 31 13:57:45.874838 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:45.874802 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" Oct 31 13:57:46.018574 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.018495 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-openshift-ca\") pod \"719ab474-6de3-473c-8dd3-6240703d5de8\" (UID: \"719ab474-6de3-473c-8dd3-6240703d5de8\") " Oct 31 13:57:46.018796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.018666 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-xks-ca\") pod \"719ab474-6de3-473c-8dd3-6240703d5de8\" (UID: \"719ab474-6de3-473c-8dd3-6240703d5de8\") " Oct 31 13:57:46.018885 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.018861 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-openshift-ca" pod "719ab474-6de3-473c-8dd3-6240703d5de8" (UID: "719ab474-6de3-473c-8dd3-6240703d5de8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:46.019017 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.018999 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719ab474-6de3-473c-8dd3-6240703d5de8-kube-api-access-7lbz8\" (UniqueName: \"kubernetes.io/projected/719ab474-6de3-473c-8dd3-6240703d5de8-kube-api-access-7lbz8\") pod \"719ab474-6de3-473c-8dd3-6240703d5de8\" (UID: \"719ab474-6de3-473c-8dd3-6240703d5de8\") " Oct 31 13:57:46.019579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.019553 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:46.020859 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.020838 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-xks-ca" pod "719ab474-6de3-473c-8dd3-6240703d5de8" (UID: "719ab474-6de3-473c-8dd3-6240703d5de8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:57:46.021218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.021191 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719ab474-6de3-473c-8dd3-6240703d5de8-kube-api-access-7lbz8" pod "719ab474-6de3-473c-8dd3-6240703d5de8" (UID: "719ab474-6de3-473c-8dd3-6240703d5de8"). InnerVolumeSpecName "kube-api-access-7lbz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:46.121448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.121412 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lbz8\" (UniqueName: \"kubernetes.io/projected/719ab474-6de3-473c-8dd3-6240703d5de8-kube-api-access-7lbz8\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:46.121448 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.121448 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/719ab474-6de3-473c-8dd3-6240703d5de8-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:46.654062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.654028 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" Oct 31 13:57:46.654062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.654055 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8" event={"ID":"719ab474-6de3-473c-8dd3-6240703d5de8","Type":"ContainerDied","Data":"9ab8c325916c8b88c80c5de8afd40989cef0d16c4df308fb47eaf4e8c9a68313"} Oct 31 13:57:46.654527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.654115 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ab8c325916c8b88c80c5de8afd40989cef0d16c4df308fb47eaf4e8c9a68313" Oct 31 13:57:46.826039 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.826008 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" Oct 31 13:57:46.832642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.832612 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-xks-ca\") pod \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\" (UID: \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\") " Oct 31 13:57:46.832916 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.832904 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-kube-api-access-sscw8\" (UniqueName: \"kubernetes.io/projected/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-kube-api-access-sscw8\") pod \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\" (UID: \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\") " Oct 31 13:57:46.832986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.832975 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-openshift-ca\") pod \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\" (UID: \"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0\") " Oct 31 13:57:46.833354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.833325 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-openshift-ca" pod "8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0" (UID: "8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:57:46.834848 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.834824 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-xks-ca" pod "8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0" (UID: "8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:57:46.835044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.835019 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-kube-api-access-sscw8" pod "8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0" (UID: "8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0"). InnerVolumeSpecName "kube-api-access-sscw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:57:46.935955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.935859 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sscw8\" (UniqueName: \"kubernetes.io/projected/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-kube-api-access-sscw8\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:46.935955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.935893 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:46.935955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:46.935904 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:57:47.665141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:47.665112 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" Oct 31 13:57:47.665141 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:47.665133 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd" event={"ID":"8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0","Type":"ContainerDied","Data":"f069862cce5a48a534abf08b529fd570aabb305f729a9a310f0f4dfe7010cdb4"} Oct 31 13:57:47.665652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:47.665180 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f069862cce5a48a534abf08b529fd570aabb305f729a9a310f0f4dfe7010cdb4" Oct 31 13:57:56.029808 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:56.029774 5532 scope.go:122] "RemoveContainer" containerID="04f26a39638f311cb8299e54e5c0aff68361a84febbba34899f58303f0b96554" Oct 31 13:57:56.041189 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:56.041159 5532 scope.go:122] "RemoveContainer" containerID="fe949cdf8b9959ad1204d1a784fb95417a95259fe741c9025b786b97d8ee965c" Oct 31 13:57:56.051325 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:56.051297 5532 scope.go:122] "RemoveContainer" containerID="e2ca391b9de05f49d333874beb26c0d10ff7279d52d9073b1b3f45489d9aa133" Oct 31 13:57:57.110156 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:57:57.110115 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:58:01.036775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:01.036696 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4"] Oct 31 13:58:01.066250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:01.066212 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a27b68e-47sr4"] Oct 31 13:58:01.787708 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:58:01.787670 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:58:02.005915 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:02.005883 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e831ca3a-c7e0-4677-91c3-d50022c2fe32" path="/var/lib/kubelet/pods/e831ca3a-c7e0-4677-91c3-d50022c2fe32/volumes" Oct 31 13:58:05.105793 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:05.105755 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg"] Oct 31 13:58:05.144136 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:05.144088 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84fa10ce-vtpsg"] Oct 31 13:58:05.557363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:05.557205 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs"] Oct 31 13:58:05.572522 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:05.572482 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5286ea-mdvbs"] Oct 31 13:58:06.005958 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:06.005924 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66852bb5-e526-4dac-a82a-a986ad2cbd5a" path="/var/lib/kubelet/pods/66852bb5-e526-4dac-a82a-a986ad2cbd5a/volumes" Oct 31 13:58:06.006484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:06.006468 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67824477-81be-4ff1-82e4-62c65528526a" path="/var/lib/kubelet/pods/67824477-81be-4ff1-82e4-62c65528526a/volumes" Oct 31 13:58:10.946902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:10.946870 5532 generic.go:356] "Generic (PLEG): container finished" podID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerID="7d48071c015ef2402428926132d720aaa6d1103f71dd7ae22840029cfb80091c" exitCode=0 Oct 31 13:58:10.947366 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:10.946939 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" event={"ID":"94c0b065-03aa-4039-beb9-1fcdc146296b","Type":"ContainerDied","Data":"7d48071c015ef2402428926132d720aaa6d1103f71dd7ae22840029cfb80091c"} Oct 31 13:58:10.947424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:10.947411 5532 scope.go:122] "RemoveContainer" containerID="7d48071c015ef2402428926132d720aaa6d1103f71dd7ae22840029cfb80091c" Oct 31 13:58:10.949537 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:10.949505 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:58:10.949680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:10.949578 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:58:10.949680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:10.949636 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:58:11.030745 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:11.030568 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9"] Oct 31 13:58:11.037393 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:11.037366 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-296ed273-t75z9"] Oct 31 13:58:12.035642 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:12.035563 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af844940-1e57-4c69-8c42-9e2dc349ae54" path="/var/lib/kubelet/pods/af844940-1e57-4c69-8c42-9e2dc349ae54/volumes" Oct 31 13:58:12.036759 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:12.036711 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" event={"ID":"94c0b065-03aa-4039-beb9-1fcdc146296b","Type":"ContainerStarted","Data":"8119b1408365d460aaf0400d0aa98647485fa81ed1fb7494280e7df5bf7bcc4e"} Oct 31 13:58:12.101718 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:12.101686 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:58:12.102680 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:12.102656 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 31 13:58:12.102738 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:12.102703 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 31 13:58:12.231283 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:12.231245 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd"] Oct 31 13:58:12.239131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:12.239040 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6a3533-2vzpd"] Oct 31 13:58:12.894711 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:12.894678 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:58:13.101691 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:13.101659 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 31 13:58:13.102124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:13.101710 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 31 13:58:13.292010 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:13.291928 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr"] Oct 31 13:58:13.292457 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:13.292423 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerName="istio-proxy" containerID="cri-o://c98a610e457f1f18f69413843b5176dd9a5107e8d6c6c71818eb17e902fe162a" gracePeriod=30 Oct 31 13:58:14.051617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:14.042216 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0" path="/var/lib/kubelet/pods/8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0/volumes" Oct 31 13:58:14.109243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:14.106051 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 31 13:58:14.109243 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:14.106132 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 31 13:58:15.032934 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:15.032900 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r"] Oct 31 13:58:15.036931 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:15.036908 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-565b4540-9667r"] Oct 31 13:58:15.102054 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:15.102023 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 31 13:58:15.102230 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:15.102082 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 31 13:58:16.005047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:16.005005 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5414af14-988d-4f31-9e65-479f5a6d07d7" path="/var/lib/kubelet/pods/5414af14-988d-4f31-9e65-479f5a6d07d7/volumes" Oct 31 13:58:16.134864 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:16.134824 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 31 13:58:16.135169 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:16.135139 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 31 13:58:17.101725 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:17.101690 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 31 13:58:17.102126 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:17.101743 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 31 13:58:18.200640 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.200604 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:58:18.203562 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.203538 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:58:18.582402 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.582377 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:58:18.648350 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.648318 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-socket\") pod \"cde88896-6051-4d05-8392-8b7eec0b2ccf\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " Oct 31 13:58:18.648556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.648357 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-envoy\") pod \"cde88896-6051-4d05-8392-8b7eec0b2ccf\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " Oct 31 13:58:18.648556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.648511 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-podinfo\") pod \"cde88896-6051-4d05-8392-8b7eec0b2ccf\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " Oct 31 13:58:18.648556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.648541 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-certs\") pod \"cde88896-6051-4d05-8392-8b7eec0b2ccf\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " Oct 31 13:58:18.650226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.649020 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-credential-socket\") pod \"cde88896-6051-4d05-8392-8b7eec0b2ccf\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " Oct 31 13:58:18.650226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.648670 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-socket" pod "cde88896-6051-4d05-8392-8b7eec0b2ccf" (UID: "cde88896-6051-4d05-8392-8b7eec0b2ccf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:58:18.650226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.648919 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-certs" pod "cde88896-6051-4d05-8392-8b7eec0b2ccf" (UID: "cde88896-6051-4d05-8392-8b7eec0b2ccf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:58:18.650226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.649112 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-token\" (UniqueName: \"kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-token\") pod \"cde88896-6051-4d05-8392-8b7eec0b2ccf\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " Oct 31 13:58:18.650226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.649204 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cde88896-6051-4d05-8392-8b7eec0b2ccf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde88896-6051-4d05-8392-8b7eec0b2ccf-istiod-ca-cert\") pod \"cde88896-6051-4d05-8392-8b7eec0b2ccf\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " Oct 31 13:58:18.650226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.649259 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-data\") pod \"cde88896-6051-4d05-8392-8b7eec0b2ccf\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " Oct 31 13:58:18.650226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.649317 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-credential-socket" pod "cde88896-6051-4d05-8392-8b7eec0b2ccf" (UID: "cde88896-6051-4d05-8392-8b7eec0b2ccf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:58:18.650226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.649338 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-kube-api-access-z2nw5\" (UniqueName: \"kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-kube-api-access-z2nw5\") pod \"cde88896-6051-4d05-8392-8b7eec0b2ccf\" (UID: \"cde88896-6051-4d05-8392-8b7eec0b2ccf\") " Oct 31 13:58:18.650226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.649644 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde88896-6051-4d05-8392-8b7eec0b2ccf-istiod-ca-cert" pod "cde88896-6051-4d05-8392-8b7eec0b2ccf" (UID: "cde88896-6051-4d05-8392-8b7eec0b2ccf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:58:18.650226 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.650168 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-data" pod "cde88896-6051-4d05-8392-8b7eec0b2ccf" (UID: "cde88896-6051-4d05-8392-8b7eec0b2ccf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:58:18.650918 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.650453 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cde88896-6051-4d05-8392-8b7eec0b2ccf-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:18.650918 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.650476 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:18.650918 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.650491 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:18.650918 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.650507 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:18.650918 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.650523 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:18.651517 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.651491 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-podinfo" pod "cde88896-6051-4d05-8392-8b7eec0b2ccf" (UID: "cde88896-6051-4d05-8392-8b7eec0b2ccf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:58:18.651873 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.651849 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-envoy" pod "cde88896-6051-4d05-8392-8b7eec0b2ccf" (UID: "cde88896-6051-4d05-8392-8b7eec0b2ccf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:58:18.652474 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.652452 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-token" pod "cde88896-6051-4d05-8392-8b7eec0b2ccf" (UID: "cde88896-6051-4d05-8392-8b7eec0b2ccf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:58:18.652693 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.652665 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-kube-api-access-z2nw5" pod "cde88896-6051-4d05-8392-8b7eec0b2ccf" (UID: "cde88896-6051-4d05-8392-8b7eec0b2ccf"). InnerVolumeSpecName "kube-api-access-z2nw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:58:18.752210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.752122 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:18.752210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.752154 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:18.752210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.752168 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2nw5\" (UniqueName: \"kubernetes.io/projected/cde88896-6051-4d05-8392-8b7eec0b2ccf-kube-api-access-z2nw5\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:18.752210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:18.752181 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cde88896-6051-4d05-8392-8b7eec0b2ccf-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:19.219203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:19.219161 5532 generic.go:356] "Generic (PLEG): container finished" podID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerID="c98a610e457f1f18f69413843b5176dd9a5107e8d6c6c71818eb17e902fe162a" exitCode=0 Oct 31 13:58:19.219740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:19.219213 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" Oct 31 13:58:19.219740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:19.219215 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" event={"ID":"cde88896-6051-4d05-8392-8b7eec0b2ccf","Type":"ContainerDied","Data":"c98a610e457f1f18f69413843b5176dd9a5107e8d6c6c71818eb17e902fe162a"} Oct 31 13:58:19.219740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:19.219286 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr" event={"ID":"cde88896-6051-4d05-8392-8b7eec0b2ccf","Type":"ContainerDied","Data":"3131c7f90cbff98dd42bd45065dc44c690d4e54ad9a2c1a4bb3694e594911767"} Oct 31 13:58:19.219740 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:19.219320 5532 scope.go:122] "RemoveContainer" containerID="c98a610e457f1f18f69413843b5176dd9a5107e8d6c6c71818eb17e902fe162a" Oct 31 13:58:19.236252 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:19.236232 5532 scope.go:122] "RemoveContainer" containerID="c98a610e457f1f18f69413843b5176dd9a5107e8d6c6c71818eb17e902fe162a" Oct 31 13:58:19.236577 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:58:19.236559 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c98a610e457f1f18f69413843b5176dd9a5107e8d6c6c71818eb17e902fe162a\": container with ID starting with c98a610e457f1f18f69413843b5176dd9a5107e8d6c6c71818eb17e902fe162a not found: ID does not exist" containerID="c98a610e457f1f18f69413843b5176dd9a5107e8d6c6c71818eb17e902fe162a" Oct 31 13:58:19.236651 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:19.236585 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c98a610e457f1f18f69413843b5176dd9a5107e8d6c6c71818eb17e902fe162a"} err="failed to get container status \"c98a610e457f1f18f69413843b5176dd9a5107e8d6c6c71818eb17e902fe162a\": rpc error: code = NotFound desc = could not find container \"c98a610e457f1f18f69413843b5176dd9a5107e8d6c6c71818eb17e902fe162a\": container with ID starting with c98a610e457f1f18f69413843b5176dd9a5107e8d6c6c71818eb17e902fe162a not found: ID does not exist" Oct 31 13:58:19.254874 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:19.254563 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr"] Oct 31 13:58:19.261431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:19.261409 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5286ea-openshift-default-646545bd4c-jq9jr"] Oct 31 13:58:20.008106 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:20.008006 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" path="/var/lib/kubelet/pods/cde88896-6051-4d05-8392-8b7eec0b2ccf/volumes" Oct 31 13:58:20.555694 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:20.555661 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g"] Oct 31 13:58:20.556316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:20.555899 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" podUID="80c8a0f2-8f1d-4733-8a30-d4852d045a0d" containerName="istio-proxy" containerID="cri-o://7efa7a86a57b783648d2d743db94d4e318ef15715fcd1892097dad47682766ee" gracePeriod=30 Oct 31 13:58:21.094769 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.094736 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2"] Oct 31 13:58:21.095587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.095565 5532 cpu_manager.go:414] "containerMap: removing container" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerName="istio-proxy" Oct 31 13:58:21.095587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.095589 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerName="istio-proxy" Oct 31 13:58:21.095742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.095627 5532 cpu_manager.go:414] "containerMap: removing container" podUID="719ab474-6de3-473c-8dd3-6240703d5de8" containerName="revoke-keys" Oct 31 13:58:21.095742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.095636 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719ab474-6de3-473c-8dd3-6240703d5de8" containerName="revoke-keys" Oct 31 13:58:21.095742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.095684 5532 cpu_manager.go:414] "containerMap: removing container" podUID="459a9c12-801e-456c-9da6-f64f297c9459" containerName="revoke-keys" Oct 31 13:58:21.095742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.095692 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="459a9c12-801e-456c-9da6-f64f297c9459" containerName="revoke-keys" Oct 31 13:58:21.095742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.095708 5532 cpu_manager.go:414] "containerMap: removing container" podUID="8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0" containerName="revoke-keys" Oct 31 13:58:21.095742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.095715 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0" containerName="revoke-keys" Oct 31 13:58:21.095944 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.095890 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="459a9c12-801e-456c-9da6-f64f297c9459" containerName="revoke-keys" Oct 31 13:58:21.095944 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.095905 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f0f0fb5-fac0-4d72-aab0-3c77dd3182c0" containerName="revoke-keys" Oct 31 13:58:21.095944 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.095926 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="cde88896-6051-4d05-8392-8b7eec0b2ccf" containerName="istio-proxy" Oct 31 13:58:21.095944 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.095937 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="719ab474-6de3-473c-8dd3-6240703d5de8" containerName="revoke-keys" Oct 31 13:58:21.099556 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.099536 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" Oct 31 13:58:21.104760 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.104739 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:58:21.104878 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.104787 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:58:21.109639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.109220 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2"] Oct 31 13:58:21.198677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.198642 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2\" (UID: \"e6f091f2-42df-4e46-947a-fc761a074ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" Oct 31 13:58:21.198854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.198688 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2\" (UID: \"e6f091f2-42df-4e46-947a-fc761a074ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" Oct 31 13:58:21.199001 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.198980 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwpdk\" (UniqueName: \"kubernetes.io/projected/e6f091f2-42df-4e46-947a-fc761a074ec9-kube-api-access-xwpdk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2\" (UID: \"e6f091f2-42df-4e46-947a-fc761a074ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" Oct 31 13:58:21.300311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.300273 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwpdk\" (UniqueName: \"kubernetes.io/projected/e6f091f2-42df-4e46-947a-fc761a074ec9-kube-api-access-xwpdk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2\" (UID: \"e6f091f2-42df-4e46-947a-fc761a074ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" Oct 31 13:58:21.300533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.300414 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2\" (UID: \"e6f091f2-42df-4e46-947a-fc761a074ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" Oct 31 13:58:21.300533 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.300448 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2\" (UID: \"e6f091f2-42df-4e46-947a-fc761a074ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" Oct 31 13:58:21.301248 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.301212 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2\" (UID: \"e6f091f2-42df-4e46-947a-fc761a074ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" Oct 31 13:58:21.303131 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.303105 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2\" (UID: \"e6f091f2-42df-4e46-947a-fc761a074ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" Oct 31 13:58:21.314579 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.314556 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwpdk\" (UniqueName: \"kubernetes.io/projected/e6f091f2-42df-4e46-947a-fc761a074ec9-kube-api-access-xwpdk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2\" (UID: \"e6f091f2-42df-4e46-947a-fc761a074ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" Oct 31 13:58:21.416094 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.416065 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" Oct 31 13:58:21.706852 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:21.706822 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2"] Oct 31 13:58:22.259562 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:22.259520 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" event={"ID":"e6f091f2-42df-4e46-947a-fc761a074ec9","Type":"ContainerStarted","Data":"5aaf419f866fba00ceda1fa4d11baac2628c546ffd7ab721af09b3001e56d54f"} Oct 31 13:58:22.259795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:22.259654 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" event={"ID":"e6f091f2-42df-4e46-947a-fc761a074ec9","Type":"ContainerStarted","Data":"943ef2c6bb2a85b7fae6368638ffad88b0ead3e02f0c840d4ef3872dabe86fe8"} Oct 31 13:58:22.281439 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:22.281365 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" podStartSLOduration=1.281350648 podStartE2EDuration="1.281350648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:58:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:58:22.278524001 +0000 UTC m=+3270.424267841" watchObservedRunningTime="2026-10-31 13:58:22.281350648 +0000 UTC m=+3270.427094475" Oct 31 13:58:23.269794 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:23.269754 5532 generic.go:356] "Generic (PLEG): container finished" podID="e6f091f2-42df-4e46-947a-fc761a074ec9" containerID="5aaf419f866fba00ceda1fa4d11baac2628c546ffd7ab721af09b3001e56d54f" exitCode=0 Oct 31 13:58:23.270213 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:23.269836 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" event={"ID":"e6f091f2-42df-4e46-947a-fc761a074ec9","Type":"ContainerDied","Data":"5aaf419f866fba00ceda1fa4d11baac2628c546ffd7ab721af09b3001e56d54f"} Oct 31 13:58:24.450401 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:24.450375 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" Oct 31 13:58:24.552354 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:24.552276 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f091f2-42df-4e46-947a-fc761a074ec9-kube-api-access-xwpdk\" (UniqueName: \"kubernetes.io/projected/e6f091f2-42df-4e46-947a-fc761a074ec9-kube-api-access-xwpdk\") pod \"e6f091f2-42df-4e46-947a-fc761a074ec9\" (UID: \"e6f091f2-42df-4e46-947a-fc761a074ec9\") " Oct 31 13:58:24.552515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:24.552363 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-openshift-ca\") pod \"e6f091f2-42df-4e46-947a-fc761a074ec9\" (UID: \"e6f091f2-42df-4e46-947a-fc761a074ec9\") " Oct 31 13:58:24.552515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:24.552409 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-xks-ca\") pod \"e6f091f2-42df-4e46-947a-fc761a074ec9\" (UID: \"e6f091f2-42df-4e46-947a-fc761a074ec9\") " Oct 31 13:58:24.552777 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:24.552746 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-openshift-ca" pod "e6f091f2-42df-4e46-947a-fc761a074ec9" (UID: "e6f091f2-42df-4e46-947a-fc761a074ec9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:58:24.553147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:24.553129 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:24.554483 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:24.554462 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-xks-ca" pod "e6f091f2-42df-4e46-947a-fc761a074ec9" (UID: "e6f091f2-42df-4e46-947a-fc761a074ec9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:58:24.554665 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:24.554644 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f091f2-42df-4e46-947a-fc761a074ec9-kube-api-access-xwpdk" pod "e6f091f2-42df-4e46-947a-fc761a074ec9" (UID: "e6f091f2-42df-4e46-947a-fc761a074ec9"). InnerVolumeSpecName "kube-api-access-xwpdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:58:24.654483 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:24.654450 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwpdk\" (UniqueName: \"kubernetes.io/projected/e6f091f2-42df-4e46-947a-fc761a074ec9-kube-api-access-xwpdk\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:24.654483 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:24.654478 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e6f091f2-42df-4e46-947a-fc761a074ec9-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:25.287025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.286996 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" Oct 31 13:58:25.287187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.287037 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2" event={"ID":"e6f091f2-42df-4e46-947a-fc761a074ec9","Type":"ContainerDied","Data":"943ef2c6bb2a85b7fae6368638ffad88b0ead3e02f0c840d4ef3872dabe86fe8"} Oct 31 13:58:25.287187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.287084 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="943ef2c6bb2a85b7fae6368638ffad88b0ead3e02f0c840d4ef3872dabe86fe8" Oct 31 13:58:25.836993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.836935 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:58:25.976162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.976123 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-kube-api-access-spdn5\" (UniqueName: \"kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-kube-api-access-spdn5\") pod \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " Oct 31 13:58:25.976368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.976189 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-envoy\") pod \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " Oct 31 13:58:25.976368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.976218 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-credential-socket\") pod \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " Oct 31 13:58:25.976368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.976266 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-podinfo\") pod \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " Oct 31 13:58:25.976368 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.976316 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-socket\") pod \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " Oct 31 13:58:25.976652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.976438 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-token\" (UniqueName: \"kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-token\") pod \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " Oct 31 13:58:25.976652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.976470 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istiod-ca-cert\") pod \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " Oct 31 13:58:25.976652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.976542 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-credential-socket" pod "80c8a0f2-8f1d-4733-8a30-d4852d045a0d" (UID: "80c8a0f2-8f1d-4733-8a30-d4852d045a0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:58:25.976902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.976673 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-socket" pod "80c8a0f2-8f1d-4733-8a30-d4852d045a0d" (UID: "80c8a0f2-8f1d-4733-8a30-d4852d045a0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:58:25.976902 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.976746 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-certs\") pod \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " Oct 31 13:58:25.977026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.976930 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-data\") pod \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\" (UID: \"80c8a0f2-8f1d-4733-8a30-d4852d045a0d\") " Oct 31 13:58:25.977026 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.977000 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-certs" pod "80c8a0f2-8f1d-4733-8a30-d4852d045a0d" (UID: "80c8a0f2-8f1d-4733-8a30-d4852d045a0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:58:25.977127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.977040 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istiod-ca-cert" pod "80c8a0f2-8f1d-4733-8a30-d4852d045a0d" (UID: "80c8a0f2-8f1d-4733-8a30-d4852d045a0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:58:25.977298 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.977274 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-data" pod "80c8a0f2-8f1d-4733-8a30-d4852d045a0d" (UID: "80c8a0f2-8f1d-4733-8a30-d4852d045a0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:58:25.977940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.977731 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:25.977940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.977747 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:25.977940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.977756 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:25.977940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.977765 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:25.977940 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.977774 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:25.978622 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.978573 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-podinfo" pod "80c8a0f2-8f1d-4733-8a30-d4852d045a0d" (UID: "80c8a0f2-8f1d-4733-8a30-d4852d045a0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:58:25.978949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.978921 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-envoy" pod "80c8a0f2-8f1d-4733-8a30-d4852d045a0d" (UID: "80c8a0f2-8f1d-4733-8a30-d4852d045a0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:58:25.979047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.978992 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-kube-api-access-spdn5" pod "80c8a0f2-8f1d-4733-8a30-d4852d045a0d" (UID: "80c8a0f2-8f1d-4733-8a30-d4852d045a0d"). InnerVolumeSpecName "kube-api-access-spdn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:58:25.979162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:25.979138 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-token" pod "80c8a0f2-8f1d-4733-8a30-d4852d045a0d" (UID: "80c8a0f2-8f1d-4733-8a30-d4852d045a0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:58:26.079470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:26.079426 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:26.079470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:26.079460 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spdn5\" (UniqueName: \"kubernetes.io/projected/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-kube-api-access-spdn5\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:26.079470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:26.079475 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:26.079775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:26.079489 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80c8a0f2-8f1d-4733-8a30-d4852d045a0d-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:26.297659 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:26.297579 5532 generic.go:356] "Generic (PLEG): container finished" podID="80c8a0f2-8f1d-4733-8a30-d4852d045a0d" containerID="7efa7a86a57b783648d2d743db94d4e318ef15715fcd1892097dad47682766ee" exitCode=0 Oct 31 13:58:26.298245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:26.297691 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" Oct 31 13:58:26.298245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:26.297689 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" event={"ID":"80c8a0f2-8f1d-4733-8a30-d4852d045a0d","Type":"ContainerDied","Data":"7efa7a86a57b783648d2d743db94d4e318ef15715fcd1892097dad47682766ee"} Oct 31 13:58:26.298245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:26.297768 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g" event={"ID":"80c8a0f2-8f1d-4733-8a30-d4852d045a0d","Type":"ContainerDied","Data":"0aa7cd7cb8bf17ab00e12000da21a9af7f4c2fed01ecf4d930b658026cbf175b"} Oct 31 13:58:26.298245 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:26.297803 5532 scope.go:122] "RemoveContainer" containerID="7efa7a86a57b783648d2d743db94d4e318ef15715fcd1892097dad47682766ee" Oct 31 13:58:26.309996 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:26.309977 5532 scope.go:122] "RemoveContainer" containerID="7efa7a86a57b783648d2d743db94d4e318ef15715fcd1892097dad47682766ee" Oct 31 13:58:26.310282 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:58:26.310253 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efa7a86a57b783648d2d743db94d4e318ef15715fcd1892097dad47682766ee\": container with ID starting with 7efa7a86a57b783648d2d743db94d4e318ef15715fcd1892097dad47682766ee not found: ID does not exist" containerID="7efa7a86a57b783648d2d743db94d4e318ef15715fcd1892097dad47682766ee" Oct 31 13:58:26.310328 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:26.310289 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efa7a86a57b783648d2d743db94d4e318ef15715fcd1892097dad47682766ee"} err="failed to get container status \"7efa7a86a57b783648d2d743db94d4e318ef15715fcd1892097dad47682766ee\": rpc error: code = NotFound desc = could not find container \"7efa7a86a57b783648d2d743db94d4e318ef15715fcd1892097dad47682766ee\": container with ID starting with 7efa7a86a57b783648d2d743db94d4e318ef15715fcd1892097dad47682766ee not found: ID does not exist" Oct 31 13:58:26.322130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:26.322099 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g"] Oct 31 13:58:26.329370 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:26.329348 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-6a3533-openshift-default-74565f4bb9-66r8g"] Oct 31 13:58:28.005252 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:28.005222 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80c8a0f2-8f1d-4733-8a30-d4852d045a0d" path="/var/lib/kubelet/pods/80c8a0f2-8f1d-4733-8a30-d4852d045a0d/volumes" Oct 31 13:58:31.764687 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:58:31.764650 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:58:40.521019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:40.520985 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2"] Oct 31 13:58:40.529033 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:40.528990 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-612b46-58xr2"] Oct 31 13:58:42.006117 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:42.006083 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f091f2-42df-4e46-947a-fc761a074ec9" path="/var/lib/kubelet/pods/e6f091f2-42df-4e46-947a-fc761a074ec9/volumes" Oct 31 13:58:46.726894 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:46.726857 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:58:50.567790 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.567742 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6"] Oct 31 13:58:50.568737 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.568713 5532 cpu_manager.go:414] "containerMap: removing container" podUID="80c8a0f2-8f1d-4733-8a30-d4852d045a0d" containerName="istio-proxy" Oct 31 13:58:50.568868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.568736 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80c8a0f2-8f1d-4733-8a30-d4852d045a0d" containerName="istio-proxy" Oct 31 13:58:50.568868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.568785 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e6f091f2-42df-4e46-947a-fc761a074ec9" containerName="revoke-keys" Oct 31 13:58:50.568868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.568793 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f091f2-42df-4e46-947a-fc761a074ec9" containerName="revoke-keys" Oct 31 13:58:50.569047 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.569034 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f091f2-42df-4e46-947a-fc761a074ec9" containerName="revoke-keys" Oct 31 13:58:50.569104 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.569069 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="80c8a0f2-8f1d-4733-8a30-d4852d045a0d" containerName="istio-proxy" Oct 31 13:58:50.574911 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.574888 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" Oct 31 13:58:50.576695 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.576667 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6"] Oct 31 13:58:50.663895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.663857 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:58:50.664095 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.663933 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:58:50.706678 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.706642 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6\" (UID: \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" Oct 31 13:58:50.706891 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.706770 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rgsv\" (UniqueName: \"kubernetes.io/projected/25ae9bf8-0079-47e4-ab79-048d3ec452a9-kube-api-access-8rgsv\") pod \"maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6\" (UID: \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" Oct 31 13:58:50.707109 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.707089 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6\" (UID: \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" Oct 31 13:58:50.808352 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.808316 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6\" (UID: \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" Oct 31 13:58:50.808548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.808366 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6\" (UID: \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" Oct 31 13:58:50.808548 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.808407 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rgsv\" (UniqueName: \"kubernetes.io/projected/25ae9bf8-0079-47e4-ab79-048d3ec452a9-kube-api-access-8rgsv\") pod \"maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6\" (UID: \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" Oct 31 13:58:50.809052 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.809030 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6\" (UID: \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" Oct 31 13:58:50.810800 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.810774 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6\" (UID: \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" Oct 31 13:58:50.822263 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.822206 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rgsv\" (UniqueName: \"kubernetes.io/projected/25ae9bf8-0079-47e4-ab79-048d3ec452a9-kube-api-access-8rgsv\") pod \"maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6\" (UID: \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" Oct 31 13:58:50.892832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:50.892797 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" Oct 31 13:58:51.209202 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:51.209166 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6"] Oct 31 13:58:51.209354 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:58:51.209306 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ae9bf8_0079_47e4_ab79_048d3ec452a9.slice/crio-36937cb0922083310c080764ea2fccf38d4b3b7a67e10d8480d48d44dc23d102 WatchSource:0}: Error finding container 36937cb0922083310c080764ea2fccf38d4b3b7a67e10d8480d48d44dc23d102: Status 404 returned error can't find the container with id 36937cb0922083310c080764ea2fccf38d4b3b7a67e10d8480d48d44dc23d102 Oct 31 13:58:51.702311 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:51.702278 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" event={"ID":"25ae9bf8-0079-47e4-ab79-048d3ec452a9","Type":"ContainerStarted","Data":"36937cb0922083310c080764ea2fccf38d4b3b7a67e10d8480d48d44dc23d102"} Oct 31 13:58:52.716508 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:52.716432 5532 generic.go:356] "Generic (PLEG): container finished" podID="25ae9bf8-0079-47e4-ab79-048d3ec452a9" containerID="49df6c2c1b521f688f76d38704c202e8e98b31e4a3ec4837bbd99f511967645b" exitCode=0 Oct 31 13:58:52.716935 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:52.716512 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" event={"ID":"25ae9bf8-0079-47e4-ab79-048d3ec452a9","Type":"ContainerDied","Data":"49df6c2c1b521f688f76d38704c202e8e98b31e4a3ec4837bbd99f511967645b"} Oct 31 13:58:53.989751 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:53.989725 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" Oct 31 13:58:54.095237 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:54.095201 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-xks-ca\") pod \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\" (UID: \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\") " Oct 31 13:58:54.095431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:54.095419 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25ae9bf8-0079-47e4-ab79-048d3ec452a9-kube-api-access-8rgsv\" (UniqueName: \"kubernetes.io/projected/25ae9bf8-0079-47e4-ab79-048d3ec452a9-kube-api-access-8rgsv\") pod \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\" (UID: \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\") " Oct 31 13:58:54.095512 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:54.095493 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-openshift-ca\") pod \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\" (UID: \"25ae9bf8-0079-47e4-ab79-048d3ec452a9\") " Oct 31 13:58:54.095961 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:54.095931 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-openshift-ca" pod "25ae9bf8-0079-47e4-ab79-048d3ec452a9" (UID: "25ae9bf8-0079-47e4-ab79-048d3ec452a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:58:54.097081 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:54.097014 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:54.097912 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:54.097883 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-xks-ca" pod "25ae9bf8-0079-47e4-ab79-048d3ec452a9" (UID: "25ae9bf8-0079-47e4-ab79-048d3ec452a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:58:54.098414 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:54.098396 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25ae9bf8-0079-47e4-ab79-048d3ec452a9-kube-api-access-8rgsv" pod "25ae9bf8-0079-47e4-ab79-048d3ec452a9" (UID: "25ae9bf8-0079-47e4-ab79-048d3ec452a9"). InnerVolumeSpecName "kube-api-access-8rgsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:58:54.198822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:54.198787 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rgsv\" (UniqueName: \"kubernetes.io/projected/25ae9bf8-0079-47e4-ab79-048d3ec452a9-kube-api-access-8rgsv\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:54.198822 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:54.198819 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25ae9bf8-0079-47e4-ab79-048d3ec452a9-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:58:54.734706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:54.734670 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" Oct 31 13:58:54.734706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:54.734692 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6" event={"ID":"25ae9bf8-0079-47e4-ab79-048d3ec452a9","Type":"ContainerDied","Data":"36937cb0922083310c080764ea2fccf38d4b3b7a67e10d8480d48d44dc23d102"} Oct 31 13:58:54.734938 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:54.734781 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36937cb0922083310c080764ea2fccf38d4b3b7a67e10d8480d48d44dc23d102" Oct 31 13:58:55.337717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:55.337683 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb"] Oct 31 13:58:55.480771 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:55.480727 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 13:58:55.480957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:55.480803 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 13:58:56.144065 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:56.144025 5532 scope.go:122] "RemoveContainer" containerID="b9844fd223b204679d886bab6fb58ea4451ee6a5caadb63bb609ea6af8cbd69b" Oct 31 13:58:58.771818 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:58.771783 5532 generic.go:356] "Generic (PLEG): container finished" podID="667e09dd-9d9e-431a-8720-c860ef93e504" containerID="83fd679e0d8219e8032d6806dedad415d23adaf87642a15d780678cd8cd4a694" exitCode=0 Oct 31 13:58:58.772266 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:58.771868 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" event={"ID":"667e09dd-9d9e-431a-8720-c860ef93e504","Type":"ContainerDied","Data":"83fd679e0d8219e8032d6806dedad415d23adaf87642a15d780678cd8cd4a694"} Oct 31 13:58:58.772411 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:58.772392 5532 scope.go:122] "RemoveContainer" containerID="83fd679e0d8219e8032d6806dedad415d23adaf87642a15d780678cd8cd4a694" Oct 31 13:58:58.774472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:58.774440 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:58:58.774562 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:58.774520 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:58:58.774617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:58.774557 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 13:58:59.782978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:59.782947 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" event={"ID":"667e09dd-9d9e-431a-8720-c860ef93e504","Type":"ContainerStarted","Data":"c4cb2e120b5e9efe4759433e6d2bfcc18dab0045f96ef9075445d6bbe045a6d4"} Oct 31 13:58:59.827996 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:59.827960 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:58:59.828325 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:59.828301 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 31 13:58:59.828442 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:58:59.828351 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 31 13:59:00.696617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:00.691049 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq"] Oct 31 13:59:00.725491 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:00.724854 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv"] Oct 31 13:59:00.829154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:00.828829 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 31 13:59:00.829154 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:00.828879 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 31 13:59:01.787954 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:01.787914 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:59:01.828142 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:01.828111 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 31 13:59:01.828319 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:01.828160 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 31 13:59:02.828378 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:02.828343 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 31 13:59:02.828796 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:02.828392 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 31 13:59:03.828166 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:03.828129 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 31 13:59:03.828389 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:03.828200 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 31 13:59:04.828197 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:04.828160 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" start-of-body= Oct 31 13:59:04.828727 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:04.828232 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.35:15021/healthz/ready\": dial tcp 10.132.1.35:15021: connect: connection refused" Oct 31 13:59:05.691589 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:05.691545 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" podUID="dcb7b65f-cd03-4643-9a7a-e246ddeafe52" containerName="payload-processing" containerID="cri-o://95ccff4e9c6f24d96e012fa019641fbfd917631ee713454d350363b7b6b04021" gracePeriod=25 Oct 31 13:59:05.725357 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:05.725318 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" podUID="93db7894-3327-4e51-8c0a-81a77124dd7a" containerName="payload-pre-processing" containerID="cri-o://2157f51a029454badb0d5b83371428263b64fbd34bd74ce856d6dfef1e9127c5" gracePeriod=25 Oct 31 13:59:05.833130 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:05.833095 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:59:05.833639 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:05.833582 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:59:05.834363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:05.834342 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 13:59:10.360236 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.360193 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" containerID="cri-o://0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe" gracePeriod=45 Oct 31 13:59:10.383134 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:10.383104 5532 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91e86d6a_00ba_4709_b2aa_0c8deda6b802.slice/crio-conmon-0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:59:10.692541 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.692509 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:59:10.802437 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.802399 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-dshm\") pod \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " Oct 31 13:59:10.802705 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.802588 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-home\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-home\") pod \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " Oct 31 13:59:10.802786 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.802736 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tmp-dir\") pod \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " Oct 31 13:59:10.803955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.802844 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tls-certs\" (UniqueName: \"kubernetes.io/secret/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tls-certs\") pod \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " Oct 31 13:59:10.803955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.802932 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91e86d6a-00ba-4709-b2aa-0c8deda6b802-kube-api-access-29wz8\" (UniqueName: \"kubernetes.io/projected/91e86d6a-00ba-4709-b2aa-0c8deda6b802-kube-api-access-29wz8\") pod \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " Oct 31 13:59:10.803955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.802974 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-model-cache\") pod \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\" (UID: \"91e86d6a-00ba-4709-b2aa-0c8deda6b802\") " Oct 31 13:59:10.803955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.803170 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-home" pod "91e86d6a-00ba-4709-b2aa-0c8deda6b802" (UID: "91e86d6a-00ba-4709-b2aa-0c8deda6b802"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:10.803955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.803756 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:10.804674 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.804444 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-model-cache" pod "91e86d6a-00ba-4709-b2aa-0c8deda6b802" (UID: "91e86d6a-00ba-4709-b2aa-0c8deda6b802"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:10.805170 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.805146 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tmp-dir" pod "91e86d6a-00ba-4709-b2aa-0c8deda6b802" (UID: "91e86d6a-00ba-4709-b2aa-0c8deda6b802"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:10.806800 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.806776 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tls-certs" pod "91e86d6a-00ba-4709-b2aa-0c8deda6b802" (UID: "91e86d6a-00ba-4709-b2aa-0c8deda6b802"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:59:10.806901 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.806776 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-dshm" pod "91e86d6a-00ba-4709-b2aa-0c8deda6b802" (UID: "91e86d6a-00ba-4709-b2aa-0c8deda6b802"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:10.807821 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.807799 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91e86d6a-00ba-4709-b2aa-0c8deda6b802-kube-api-access-29wz8" pod "91e86d6a-00ba-4709-b2aa-0c8deda6b802" (UID: "91e86d6a-00ba-4709-b2aa-0c8deda6b802"). InnerVolumeSpecName "kube-api-access-29wz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:10.889360 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.889326 5532 generic.go:356] "Generic (PLEG): container finished" podID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerID="0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe" exitCode=0 Oct 31 13:59:10.889547 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.889374 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" Oct 31 13:59:10.889547 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.889383 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" event={"ID":"91e86d6a-00ba-4709-b2aa-0c8deda6b802","Type":"ContainerDied","Data":"0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe"} Oct 31 13:59:10.889547 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.889420 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb" event={"ID":"91e86d6a-00ba-4709-b2aa-0c8deda6b802","Type":"ContainerDied","Data":"d906cc8a1825989d4e9221b6d2969a4b599c9471e136562588b33a77af4b253c"} Oct 31 13:59:10.889547 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.889436 5532 scope.go:122] "RemoveContainer" containerID="0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe" Oct 31 13:59:10.905498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.905468 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:10.905498 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.905497 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29wz8\" (UniqueName: \"kubernetes.io/projected/91e86d6a-00ba-4709-b2aa-0c8deda6b802-kube-api-access-29wz8\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:10.905717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.905507 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:10.905717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.905516 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:10.905717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.905525 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91e86d6a-00ba-4709-b2aa-0c8deda6b802-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:10.906542 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.906523 5532 scope.go:122] "RemoveContainer" containerID="0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe" Oct 31 13:59:10.906930 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:10.906899 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe\": container with ID starting with 0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe not found: ID does not exist" containerID="0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe" Oct 31 13:59:10.907064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.906942 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe"} err="failed to get container status \"0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe\": rpc error: code = NotFound desc = could not find container \"0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe\": container with ID starting with 0a14dc5901b1aee94dbbb66f968a2d5d169c7a836bbc88f345224b4570ffdabe not found: ID does not exist" Oct 31 13:59:10.920034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.920007 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb"] Oct 31 13:59:10.927563 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:10.927538 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-7b3ddd/distinct-2-w0-7b3ddd-kserve-7b84b96b7b-5kwnb"] Oct 31 13:59:12.006824 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:12.006790 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" path="/var/lib/kubelet/pods/91e86d6a-00ba-4709-b2aa-0c8deda6b802/volumes" Oct 31 13:59:13.789317 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:13.789264 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" podUID="dcb7b65f-cd03-4643-9a7a-e246ddeafe52" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:59:13.789782 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:13.789279 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" podUID="93db7894-3327-4e51-8c0a-81a77124dd7a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:59:15.135361 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.135323 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6"] Oct 31 13:59:15.136120 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.136104 5532 cpu_manager.go:414] "containerMap: removing container" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" Oct 31 13:59:15.136171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.136121 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" Oct 31 13:59:15.136171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.136130 5532 cpu_manager.go:414] "containerMap: removing container" podUID="25ae9bf8-0079-47e4-ab79-048d3ec452a9" containerName="revoke-keys" Oct 31 13:59:15.136171 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.136135 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25ae9bf8-0079-47e4-ab79-048d3ec452a9" containerName="revoke-keys" Oct 31 13:59:15.136271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.136251 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="91e86d6a-00ba-4709-b2aa-0c8deda6b802" containerName="main" Oct 31 13:59:15.136271 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.136258 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="25ae9bf8-0079-47e4-ab79-048d3ec452a9" containerName="revoke-keys" Oct 31 13:59:15.141430 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.141412 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" Oct 31 13:59:15.147503 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.147475 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:59:15.147807 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.147788 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:59:15.150132 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.150101 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6"] Oct 31 13:59:15.279529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.279493 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snjfh\" (UniqueName: \"kubernetes.io/projected/971eccfa-4d3c-4154-9870-388b6ca5e9f8-kube-api-access-snjfh\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6\" (UID: \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" Oct 31 13:59:15.279806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.279668 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6\" (UID: \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" Oct 31 13:59:15.279903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.279803 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6\" (UID: \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" Oct 31 13:59:15.381105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.380994 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6\" (UID: \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" Oct 31 13:59:15.381404 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.381384 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snjfh\" (UniqueName: \"kubernetes.io/projected/971eccfa-4d3c-4154-9870-388b6ca5e9f8-kube-api-access-snjfh\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6\" (UID: \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" Oct 31 13:59:15.381531 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.381513 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6\" (UID: \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" Oct 31 13:59:15.381696 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.381670 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6\" (UID: \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" Oct 31 13:59:15.383989 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.383965 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6\" (UID: \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" Oct 31 13:59:15.394387 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.394369 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snjfh\" (UniqueName: \"kubernetes.io/projected/971eccfa-4d3c-4154-9870-388b6ca5e9f8-kube-api-access-snjfh\") pod \"maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6\" (UID: \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" Oct 31 13:59:15.461022 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.460991 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" Oct 31 13:59:15.666992 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.666909 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht"] Oct 31 13:59:15.673034 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.673006 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" Oct 31 13:59:15.673294 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.673271 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs"] Oct 31 13:59:15.678933 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.678906 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" Oct 31 13:59:15.679644 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.679619 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht"] Oct 31 13:59:15.685946 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.685916 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs"] Oct 31 13:59:15.754134 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.754108 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6"] Oct 31 13:59:15.755170 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:59:15.755143 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod971eccfa_4d3c_4154_9870_388b6ca5e9f8.slice/crio-c08ea2206763492015722eff21b2ebb45ec5d56186aecf3f8debfd3dabd61c0e WatchSource:0}: Error finding container c08ea2206763492015722eff21b2ebb45ec5d56186aecf3f8debfd3dabd61c0e: Status 404 returned error can't find the container with id c08ea2206763492015722eff21b2ebb45ec5d56186aecf3f8debfd3dabd61c0e Oct 31 13:59:15.787127 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.787099 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0890bf9-w97zs\" (UID: \"ef9796f9-cb17-4345-98ff-e7de08c1309f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" Oct 31 13:59:15.787227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.787170 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-809457e8-vrxht\" (UID: \"c7500c1d-c528-444e-8160-6ecf4244c146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" Oct 31 13:59:15.787356 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.787237 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgjd8\" (UniqueName: \"kubernetes.io/projected/ef9796f9-cb17-4345-98ff-e7de08c1309f-kube-api-access-kgjd8\") pod \"maas-api-revoke-sub-e0890bf9-w97zs\" (UID: \"ef9796f9-cb17-4345-98ff-e7de08c1309f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" Oct 31 13:59:15.787434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.787373 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0890bf9-w97zs\" (UID: \"ef9796f9-cb17-4345-98ff-e7de08c1309f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" Oct 31 13:59:15.787434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.787417 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsggb\" (UniqueName: \"kubernetes.io/projected/c7500c1d-c528-444e-8160-6ecf4244c146-kube-api-access-nsggb\") pod \"maas-api-revoke-sub-809457e8-vrxht\" (UID: \"c7500c1d-c528-444e-8160-6ecf4244c146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" Oct 31 13:59:15.787515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.787445 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-809457e8-vrxht\" (UID: \"c7500c1d-c528-444e-8160-6ecf4244c146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" Oct 31 13:59:15.889494 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.889464 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0890bf9-w97zs\" (UID: \"ef9796f9-cb17-4345-98ff-e7de08c1309f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" Oct 31 13:59:15.889715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.889526 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-809457e8-vrxht\" (UID: \"c7500c1d-c528-444e-8160-6ecf4244c146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" Oct 31 13:59:15.889715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.889561 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgjd8\" (UniqueName: \"kubernetes.io/projected/ef9796f9-cb17-4345-98ff-e7de08c1309f-kube-api-access-kgjd8\") pod \"maas-api-revoke-sub-e0890bf9-w97zs\" (UID: \"ef9796f9-cb17-4345-98ff-e7de08c1309f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" Oct 31 13:59:15.889715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.889663 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0890bf9-w97zs\" (UID: \"ef9796f9-cb17-4345-98ff-e7de08c1309f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" Oct 31 13:59:15.889715 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.889698 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsggb\" (UniqueName: \"kubernetes.io/projected/c7500c1d-c528-444e-8160-6ecf4244c146-kube-api-access-nsggb\") pod \"maas-api-revoke-sub-809457e8-vrxht\" (UID: \"c7500c1d-c528-444e-8160-6ecf4244c146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" Oct 31 13:59:15.889983 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.889811 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-809457e8-vrxht\" (UID: \"c7500c1d-c528-444e-8160-6ecf4244c146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" Oct 31 13:59:15.890352 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.890327 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-809457e8-vrxht\" (UID: \"c7500c1d-c528-444e-8160-6ecf4244c146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" Oct 31 13:59:15.890459 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.890362 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0890bf9-w97zs\" (UID: \"ef9796f9-cb17-4345-98ff-e7de08c1309f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" Oct 31 13:59:15.891980 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.891962 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0890bf9-w97zs\" (UID: \"ef9796f9-cb17-4345-98ff-e7de08c1309f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" Oct 31 13:59:15.892218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.892199 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-xks-ca\") pod \"maas-api-revoke-sub-809457e8-vrxht\" (UID: \"c7500c1d-c528-444e-8160-6ecf4244c146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" Oct 31 13:59:15.903246 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.903219 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgjd8\" (UniqueName: \"kubernetes.io/projected/ef9796f9-cb17-4345-98ff-e7de08c1309f-kube-api-access-kgjd8\") pod \"maas-api-revoke-sub-e0890bf9-w97zs\" (UID: \"ef9796f9-cb17-4345-98ff-e7de08c1309f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" Oct 31 13:59:15.903446 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.903425 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsggb\" (UniqueName: \"kubernetes.io/projected/c7500c1d-c528-444e-8160-6ecf4244c146-kube-api-access-nsggb\") pod \"maas-api-revoke-sub-809457e8-vrxht\" (UID: \"c7500c1d-c528-444e-8160-6ecf4244c146\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" Oct 31 13:59:15.952332 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.952253 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" event={"ID":"971eccfa-4d3c-4154-9870-388b6ca5e9f8","Type":"ContainerStarted","Data":"c08ea2206763492015722eff21b2ebb45ec5d56186aecf3f8debfd3dabd61c0e"} Oct 31 13:59:15.999274 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:15.999248 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" Oct 31 13:59:16.008048 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:16.008021 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" Oct 31 13:59:16.296854 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:16.296822 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht"] Oct 31 13:59:16.298820 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:59:16.298785 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7500c1d_c528_444e_8160_6ecf4244c146.slice/crio-91e4c72eb8866777bd61b25d4902d67807f659e865fb34ec29e5f29b1f94c185 WatchSource:0}: Error finding container 91e4c72eb8866777bd61b25d4902d67807f659e865fb34ec29e5f29b1f94c185: Status 404 returned error can't find the container with id 91e4c72eb8866777bd61b25d4902d67807f659e865fb34ec29e5f29b1f94c185 Oct 31 13:59:16.399993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:16.399964 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs"] Oct 31 13:59:16.400707 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:59:16.400678 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9796f9_cb17_4345_98ff_e7de08c1309f.slice/crio-e3b9bf297e6e5ddcbca73b5d437829a0e84854c13ef627535833ecac6eff288b WatchSource:0}: Error finding container e3b9bf297e6e5ddcbca73b5d437829a0e84854c13ef627535833ecac6eff288b: Status 404 returned error can't find the container with id e3b9bf297e6e5ddcbca73b5d437829a0e84854c13ef627535833ecac6eff288b Oct 31 13:59:16.756101 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:16.756062 5532 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9796f9_cb17_4345_98ff_e7de08c1309f.slice/crio-faecc4f125de25fe04ad568f2c48a46c59487aa569b39ff2faa4463714763bd3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:59:16.762680 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:16.762638 5532 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef9796f9_cb17_4345_98ff_e7de08c1309f.slice/crio-faecc4f125de25fe04ad568f2c48a46c59487aa569b39ff2faa4463714763bd3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 13:59:17.021066 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:17.021021 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" event={"ID":"ef9796f9-cb17-4345-98ff-e7de08c1309f","Type":"ContainerStarted","Data":"e3b9bf297e6e5ddcbca73b5d437829a0e84854c13ef627535833ecac6eff288b"} Oct 31 13:59:17.024855 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:17.024817 5532 generic.go:356] "Generic (PLEG): container finished" podID="971eccfa-4d3c-4154-9870-388b6ca5e9f8" containerID="ba6514182c916cb73079ac6b89aaeb6e2e96d2616073c9d8fdc70de71408369a" exitCode=0 Oct 31 13:59:17.024986 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:17.024961 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" event={"ID":"971eccfa-4d3c-4154-9870-388b6ca5e9f8","Type":"ContainerDied","Data":"ba6514182c916cb73079ac6b89aaeb6e2e96d2616073c9d8fdc70de71408369a"} Oct 31 13:59:17.028287 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:17.028267 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" event={"ID":"c7500c1d-c528-444e-8160-6ecf4244c146","Type":"ContainerStarted","Data":"0f93e3640e20d465d7658b19eec7844408b7b5dc01901708deb0fe93d6fd397f"} Oct 31 13:59:17.028431 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:17.028411 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" event={"ID":"c7500c1d-c528-444e-8160-6ecf4244c146","Type":"ContainerStarted","Data":"91e4c72eb8866777bd61b25d4902d67807f659e865fb34ec29e5f29b1f94c185"} Oct 31 13:59:17.121629 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:17.118812 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" podStartSLOduration=2.118790823 podStartE2EDuration="2.118790823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:59:17.092434421 +0000 UTC m=+3325.238178253" watchObservedRunningTime="2026-10-31 13:59:17.118790823 +0000 UTC m=+3325.264534652" Oct 31 13:59:18.043172 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.043138 5532 generic.go:356] "Generic (PLEG): container finished" podID="ef9796f9-cb17-4345-98ff-e7de08c1309f" containerID="faecc4f125de25fe04ad568f2c48a46c59487aa569b39ff2faa4463714763bd3" exitCode=0 Oct 31 13:59:18.043801 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.043226 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" event={"ID":"ef9796f9-cb17-4345-98ff-e7de08c1309f","Type":"ContainerDied","Data":"faecc4f125de25fe04ad568f2c48a46c59487aa569b39ff2faa4463714763bd3"} Oct 31 13:59:18.045210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.045186 5532 generic.go:356] "Generic (PLEG): container finished" podID="c7500c1d-c528-444e-8160-6ecf4244c146" containerID="0f93e3640e20d465d7658b19eec7844408b7b5dc01901708deb0fe93d6fd397f" exitCode=0 Oct 31 13:59:18.045337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.045264 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" event={"ID":"c7500c1d-c528-444e-8160-6ecf4244c146","Type":"ContainerDied","Data":"0f93e3640e20d465d7658b19eec7844408b7b5dc01901708deb0fe93d6fd397f"} Oct 31 13:59:18.244250 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.244226 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" Oct 31 13:59:18.387527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.387434 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-openshift-ca\") pod \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\" (UID: \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\") " Oct 31 13:59:18.387527 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.387507 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/971eccfa-4d3c-4154-9870-388b6ca5e9f8-kube-api-access-snjfh\" (UniqueName: \"kubernetes.io/projected/971eccfa-4d3c-4154-9870-388b6ca5e9f8-kube-api-access-snjfh\") pod \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\" (UID: \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\") " Oct 31 13:59:18.387833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.387588 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-xks-ca\") pod \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\" (UID: \"971eccfa-4d3c-4154-9870-388b6ca5e9f8\") " Oct 31 13:59:18.387898 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.387846 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-openshift-ca" pod "971eccfa-4d3c-4154-9870-388b6ca5e9f8" (UID: "971eccfa-4d3c-4154-9870-388b6ca5e9f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:18.388300 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.388277 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:18.389804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.389774 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-xks-ca" pod "971eccfa-4d3c-4154-9870-388b6ca5e9f8" (UID: "971eccfa-4d3c-4154-9870-388b6ca5e9f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:59:18.389804 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.389783 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/971eccfa-4d3c-4154-9870-388b6ca5e9f8-kube-api-access-snjfh" pod "971eccfa-4d3c-4154-9870-388b6ca5e9f8" (UID: "971eccfa-4d3c-4154-9870-388b6ca5e9f8"). InnerVolumeSpecName "kube-api-access-snjfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:18.489561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.489530 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snjfh\" (UniqueName: \"kubernetes.io/projected/971eccfa-4d3c-4154-9870-388b6ca5e9f8-kube-api-access-snjfh\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:18.489561 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:18.489560 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/971eccfa-4d3c-4154-9870-388b6ca5e9f8-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:19.054975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.054934 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" event={"ID":"971eccfa-4d3c-4154-9870-388b6ca5e9f8","Type":"ContainerDied","Data":"c08ea2206763492015722eff21b2ebb45ec5d56186aecf3f8debfd3dabd61c0e"} Oct 31 13:59:19.054975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.054956 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6" Oct 31 13:59:19.055437 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.054985 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c08ea2206763492015722eff21b2ebb45ec5d56186aecf3f8debfd3dabd61c0e" Oct 31 13:59:19.311424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.311365 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" Oct 31 13:59:19.314008 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.313988 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" Oct 31 13:59:19.404615 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.404567 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7500c1d-c528-444e-8160-6ecf4244c146-kube-api-access-nsggb\" (UniqueName: \"kubernetes.io/projected/c7500c1d-c528-444e-8160-6ecf4244c146-kube-api-access-nsggb\") pod \"c7500c1d-c528-444e-8160-6ecf4244c146\" (UID: \"c7500c1d-c528-444e-8160-6ecf4244c146\") " Oct 31 13:59:19.404815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.404674 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-openshift-ca\") pod \"c7500c1d-c528-444e-8160-6ecf4244c146\" (UID: \"c7500c1d-c528-444e-8160-6ecf4244c146\") " Oct 31 13:59:19.404815 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.404805 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef9796f9-cb17-4345-98ff-e7de08c1309f-kube-api-access-kgjd8\" (UniqueName: \"kubernetes.io/projected/ef9796f9-cb17-4345-98ff-e7de08c1309f-kube-api-access-kgjd8\") pod \"ef9796f9-cb17-4345-98ff-e7de08c1309f\" (UID: \"ef9796f9-cb17-4345-98ff-e7de08c1309f\") " Oct 31 13:59:19.404956 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.404926 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-xks-ca\") pod \"c7500c1d-c528-444e-8160-6ecf4244c146\" (UID: \"c7500c1d-c528-444e-8160-6ecf4244c146\") " Oct 31 13:59:19.405019 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.404987 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-openshift-ca\") pod \"ef9796f9-cb17-4345-98ff-e7de08c1309f\" (UID: \"ef9796f9-cb17-4345-98ff-e7de08c1309f\") " Oct 31 13:59:19.405074 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.405029 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-xks-ca\") pod \"ef9796f9-cb17-4345-98ff-e7de08c1309f\" (UID: \"ef9796f9-cb17-4345-98ff-e7de08c1309f\") " Oct 31 13:59:19.406043 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.406011 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-openshift-ca" pod "ef9796f9-cb17-4345-98ff-e7de08c1309f" (UID: "ef9796f9-cb17-4345-98ff-e7de08c1309f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:19.406399 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.406367 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-openshift-ca" pod "c7500c1d-c528-444e-8160-6ecf4244c146" (UID: "c7500c1d-c528-444e-8160-6ecf4244c146"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:19.407450 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.407420 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7500c1d-c528-444e-8160-6ecf4244c146-kube-api-access-nsggb" pod "c7500c1d-c528-444e-8160-6ecf4244c146" (UID: "c7500c1d-c528-444e-8160-6ecf4244c146"). InnerVolumeSpecName "kube-api-access-nsggb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:19.408558 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.408529 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef9796f9-cb17-4345-98ff-e7de08c1309f-kube-api-access-kgjd8" pod "ef9796f9-cb17-4345-98ff-e7de08c1309f" (UID: "ef9796f9-cb17-4345-98ff-e7de08c1309f"). InnerVolumeSpecName "kube-api-access-kgjd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:19.408736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.408644 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-xks-ca" pod "ef9796f9-cb17-4345-98ff-e7de08c1309f" (UID: "ef9796f9-cb17-4345-98ff-e7de08c1309f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:59:19.408736 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.408696 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-xks-ca" pod "c7500c1d-c528-444e-8160-6ecf4244c146" (UID: "c7500c1d-c528-444e-8160-6ecf4244c146"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:59:19.507208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.507172 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgjd8\" (UniqueName: \"kubernetes.io/projected/ef9796f9-cb17-4345-98ff-e7de08c1309f-kube-api-access-kgjd8\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:19.507208 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.507206 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:19.507415 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.507217 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:19.507415 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.507226 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef9796f9-cb17-4345-98ff-e7de08c1309f-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:19.507415 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.507235 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsggb\" (UniqueName: \"kubernetes.io/projected/c7500c1d-c528-444e-8160-6ecf4244c146-kube-api-access-nsggb\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:19.507415 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:19.507244 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7500c1d-c528-444e-8160-6ecf4244c146-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:20.066192 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:20.066162 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" Oct 31 13:59:20.066623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:20.066187 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs" event={"ID":"ef9796f9-cb17-4345-98ff-e7de08c1309f","Type":"ContainerDied","Data":"e3b9bf297e6e5ddcbca73b5d437829a0e84854c13ef627535833ecac6eff288b"} Oct 31 13:59:20.066623 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:20.066236 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3b9bf297e6e5ddcbca73b5d437829a0e84854c13ef627535833ecac6eff288b" Oct 31 13:59:20.067806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:20.067782 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" event={"ID":"c7500c1d-c528-444e-8160-6ecf4244c146","Type":"ContainerDied","Data":"91e4c72eb8866777bd61b25d4902d67807f659e865fb34ec29e5f29b1f94c185"} Oct 31 13:59:20.067913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:20.067805 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht" Oct 31 13:59:20.067913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:20.067828 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91e4c72eb8866777bd61b25d4902d67807f659e865fb34ec29e5f29b1f94c185" Oct 31 13:59:22.049505 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:22.049472 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf_51e61ea6-8c12-4113-a371-a245526ed535/payload-processing/0.log" Oct 31 13:59:23.454636 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.454566 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7"] Oct 31 13:59:23.455572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.455549 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c7500c1d-c528-444e-8160-6ecf4244c146" containerName="revoke-keys" Oct 31 13:59:23.455572 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.455572 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7500c1d-c528-444e-8160-6ecf4244c146" containerName="revoke-keys" Oct 31 13:59:23.455812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.455646 5532 cpu_manager.go:414] "containerMap: removing container" podUID="971eccfa-4d3c-4154-9870-388b6ca5e9f8" containerName="revoke-keys" Oct 31 13:59:23.455812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.455654 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="971eccfa-4d3c-4154-9870-388b6ca5e9f8" containerName="revoke-keys" Oct 31 13:59:23.455812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.455691 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ef9796f9-cb17-4345-98ff-e7de08c1309f" containerName="revoke-keys" Oct 31 13:59:23.455812 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.455699 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef9796f9-cb17-4345-98ff-e7de08c1309f" containerName="revoke-keys" Oct 31 13:59:23.456024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.455897 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="971eccfa-4d3c-4154-9870-388b6ca5e9f8" containerName="revoke-keys" Oct 31 13:59:23.456024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.455917 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef9796f9-cb17-4345-98ff-e7de08c1309f" containerName="revoke-keys" Oct 31 13:59:23.456024 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.455936 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7500c1d-c528-444e-8160-6ecf4244c146" containerName="revoke-keys" Oct 31 13:59:23.459833 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.459812 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" Oct 31 13:59:23.465161 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.465138 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:59:23.465329 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.465250 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:59:23.469849 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.469826 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7"] Oct 31 13:59:23.571537 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.571501 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d929k\" (UniqueName: \"kubernetes.io/projected/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-kube-api-access-d929k\") pod \"maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7\" (UID: \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" Oct 31 13:59:23.571742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.571555 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7\" (UID: \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" Oct 31 13:59:23.571742 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.571721 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7\" (UID: \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" Oct 31 13:59:23.673076 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.673044 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d929k\" (UniqueName: \"kubernetes.io/projected/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-kube-api-access-d929k\") pod \"maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7\" (UID: \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" Oct 31 13:59:23.673288 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.673093 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7\" (UID: \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" Oct 31 13:59:23.673288 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.673159 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7\" (UID: \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" Oct 31 13:59:23.673832 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.673806 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7\" (UID: \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" Oct 31 13:59:23.675461 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.675442 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7\" (UID: \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" Oct 31 13:59:23.686885 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.686857 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d929k\" (UniqueName: \"kubernetes.io/projected/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-kube-api-access-d929k\") pod \"maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7\" (UID: \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" Oct 31 13:59:23.776886 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.776785 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" Oct 31 13:59:23.789798 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.789756 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" podUID="dcb7b65f-cd03-4643-9a7a-e246ddeafe52" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:59:23.790064 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:23.790038 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" podUID="93db7894-3327-4e51-8c0a-81a77124dd7a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 13:59:24.064345 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:59:24.064307 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cefbe6c_67a4_4d0b_9246_29a382b4bd96.slice/crio-53296acfeca2db38dd9f6de6eb5c3f4bf678ca09776b6c7e78c4148fe2067c0c WatchSource:0}: Error finding container 53296acfeca2db38dd9f6de6eb5c3f4bf678ca09776b6c7e78c4148fe2067c0c: Status 404 returned error can't find the container with id 53296acfeca2db38dd9f6de6eb5c3f4bf678ca09776b6c7e78c4148fe2067c0c Oct 31 13:59:24.064868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:24.064844 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7"] Oct 31 13:59:24.126144 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:24.126115 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" event={"ID":"4cefbe6c-67a4-4d0b-9246-29a382b4bd96","Type":"ContainerStarted","Data":"53296acfeca2db38dd9f6de6eb5c3f4bf678ca09776b6c7e78c4148fe2067c0c"} Oct 31 13:59:25.136244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:25.136207 5532 generic.go:356] "Generic (PLEG): container finished" podID="4cefbe6c-67a4-4d0b-9246-29a382b4bd96" containerID="dffa9693f69f07049995d1663e40ac14cf4c3d8c689c5104c7889d79302dcc6f" exitCode=0 Oct 31 13:59:25.136717 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:25.136293 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" event={"ID":"4cefbe6c-67a4-4d0b-9246-29a382b4bd96","Type":"ContainerDied","Data":"dffa9693f69f07049995d1663e40ac14cf4c3d8c689c5104c7889d79302dcc6f"} Oct 31 13:59:26.030376 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.030351 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" Oct 31 13:59:26.070775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.070745 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" Oct 31 13:59:26.158419 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.158383 5532 generic.go:356] "Generic (PLEG): container finished" podID="93db7894-3327-4e51-8c0a-81a77124dd7a" containerID="2157f51a029454badb0d5b83371428263b64fbd34bd74ce856d6dfef1e9127c5" exitCode=0 Oct 31 13:59:26.158905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.158526 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" Oct 31 13:59:26.158905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.158525 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" event={"ID":"93db7894-3327-4e51-8c0a-81a77124dd7a","Type":"ContainerDied","Data":"2157f51a029454badb0d5b83371428263b64fbd34bd74ce856d6dfef1e9127c5"} Oct 31 13:59:26.158905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.158691 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv" event={"ID":"93db7894-3327-4e51-8c0a-81a77124dd7a","Type":"ContainerDied","Data":"01307333f4cee48d59ecdb2d5c91e8709282036597a2a19303950301633038f2"} Oct 31 13:59:26.158905 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.158721 5532 scope.go:122] "RemoveContainer" containerID="2157f51a029454badb0d5b83371428263b64fbd34bd74ce856d6dfef1e9127c5" Oct 31 13:59:26.160367 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.160344 5532 generic.go:356] "Generic (PLEG): container finished" podID="dcb7b65f-cd03-4643-9a7a-e246ddeafe52" containerID="95ccff4e9c6f24d96e012fa019641fbfd917631ee713454d350363b7b6b04021" exitCode=0 Oct 31 13:59:26.161210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.160730 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" Oct 31 13:59:26.161210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.160906 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" event={"ID":"dcb7b65f-cd03-4643-9a7a-e246ddeafe52","Type":"ContainerDied","Data":"95ccff4e9c6f24d96e012fa019641fbfd917631ee713454d350363b7b6b04021"} Oct 31 13:59:26.161210 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.160947 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq" event={"ID":"dcb7b65f-cd03-4643-9a7a-e246ddeafe52","Type":"ContainerDied","Data":"148f0c190da84b46f408ef4a362bd4644ab79b82e1c68321ca01dc0abdbb12a2"} Oct 31 13:59:26.175084 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.175055 5532 scope.go:122] "RemoveContainer" containerID="2157f51a029454badb0d5b83371428263b64fbd34bd74ce856d6dfef1e9127c5" Oct 31 13:59:26.175646 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:26.175575 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2157f51a029454badb0d5b83371428263b64fbd34bd74ce856d6dfef1e9127c5\": container with ID starting with 2157f51a029454badb0d5b83371428263b64fbd34bd74ce856d6dfef1e9127c5 not found: ID does not exist" containerID="2157f51a029454badb0d5b83371428263b64fbd34bd74ce856d6dfef1e9127c5" Oct 31 13:59:26.175781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.175662 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2157f51a029454badb0d5b83371428263b64fbd34bd74ce856d6dfef1e9127c5"} err="failed to get container status \"2157f51a029454badb0d5b83371428263b64fbd34bd74ce856d6dfef1e9127c5\": rpc error: code = NotFound desc = could not find container \"2157f51a029454badb0d5b83371428263b64fbd34bd74ce856d6dfef1e9127c5\": container with ID starting with 2157f51a029454badb0d5b83371428263b64fbd34bd74ce856d6dfef1e9127c5 not found: ID does not exist" Oct 31 13:59:26.175781 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.175689 5532 scope.go:122] "RemoveContainer" containerID="95ccff4e9c6f24d96e012fa019641fbfd917631ee713454d350363b7b6b04021" Oct 31 13:59:26.192359 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.192339 5532 scope.go:122] "RemoveContainer" containerID="95ccff4e9c6f24d96e012fa019641fbfd917631ee713454d350363b7b6b04021" Oct 31 13:59:26.192783 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:26.192736 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95ccff4e9c6f24d96e012fa019641fbfd917631ee713454d350363b7b6b04021\": container with ID starting with 95ccff4e9c6f24d96e012fa019641fbfd917631ee713454d350363b7b6b04021 not found: ID does not exist" containerID="95ccff4e9c6f24d96e012fa019641fbfd917631ee713454d350363b7b6b04021" Oct 31 13:59:26.192783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.192763 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95ccff4e9c6f24d96e012fa019641fbfd917631ee713454d350363b7b6b04021"} err="failed to get container status \"95ccff4e9c6f24d96e012fa019641fbfd917631ee713454d350363b7b6b04021\": rpc error: code = NotFound desc = could not find container \"95ccff4e9c6f24d96e012fa019641fbfd917631ee713454d350363b7b6b04021\": container with ID starting with 95ccff4e9c6f24d96e012fa019641fbfd917631ee713454d350363b7b6b04021 not found: ID does not exist" Oct 31 13:59:26.213551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.213481 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-plugins-config-volume\") pod \"dcb7b65f-cd03-4643-9a7a-e246ddeafe52\" (UID: \"dcb7b65f-cd03-4643-9a7a-e246ddeafe52\") " Oct 31 13:59:26.213551 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.213550 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-kube-api-access-fs7kn\" (UniqueName: \"kubernetes.io/projected/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-kube-api-access-fs7kn\") pod \"dcb7b65f-cd03-4643-9a7a-e246ddeafe52\" (UID: \"dcb7b65f-cd03-4643-9a7a-e246ddeafe52\") " Oct 31 13:59:26.213783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.213620 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93db7894-3327-4e51-8c0a-81a77124dd7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93db7894-3327-4e51-8c0a-81a77124dd7a-plugins-config-volume\") pod \"93db7894-3327-4e51-8c0a-81a77124dd7a\" (UID: \"93db7894-3327-4e51-8c0a-81a77124dd7a\") " Oct 31 13:59:26.213783 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.213650 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93db7894-3327-4e51-8c0a-81a77124dd7a-kube-api-access-np7rq\" (UniqueName: \"kubernetes.io/projected/93db7894-3327-4e51-8c0a-81a77124dd7a-kube-api-access-np7rq\") pod \"93db7894-3327-4e51-8c0a-81a77124dd7a\" (UID: \"93db7894-3327-4e51-8c0a-81a77124dd7a\") " Oct 31 13:59:26.213957 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.213915 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-plugins-config-volume" pod "dcb7b65f-cd03-4643-9a7a-e246ddeafe52" (UID: "dcb7b65f-cd03-4643-9a7a-e246ddeafe52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:26.214268 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.214021 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93db7894-3327-4e51-8c0a-81a77124dd7a-plugins-config-volume" pod "93db7894-3327-4e51-8c0a-81a77124dd7a" (UID: "93db7894-3327-4e51-8c0a-81a77124dd7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:26.214470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.214411 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93db7894-3327-4e51-8c0a-81a77124dd7a-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:26.214470 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.214442 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:26.215877 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.215849 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93db7894-3327-4e51-8c0a-81a77124dd7a-kube-api-access-np7rq" pod "93db7894-3327-4e51-8c0a-81a77124dd7a" (UID: "93db7894-3327-4e51-8c0a-81a77124dd7a"). InnerVolumeSpecName "kube-api-access-np7rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:26.216534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.216514 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-kube-api-access-fs7kn" pod "dcb7b65f-cd03-4643-9a7a-e246ddeafe52" (UID: "dcb7b65f-cd03-4643-9a7a-e246ddeafe52"). InnerVolumeSpecName "kube-api-access-fs7kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:26.316410 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.316380 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs7kn\" (UniqueName: \"kubernetes.io/projected/dcb7b65f-cd03-4643-9a7a-e246ddeafe52-kube-api-access-fs7kn\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:26.316410 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.316408 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np7rq\" (UniqueName: \"kubernetes.io/projected/93db7894-3327-4e51-8c0a-81a77124dd7a-kube-api-access-np7rq\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:26.317795 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.317778 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" Oct 31 13:59:26.417246 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.417214 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-openshift-ca\") pod \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\" (UID: \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\") " Oct 31 13:59:26.417424 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.417307 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-kube-api-access-d929k\" (UniqueName: \"kubernetes.io/projected/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-kube-api-access-d929k\") pod \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\" (UID: \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\") " Oct 31 13:59:26.417529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.417508 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-xks-ca\") pod \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\" (UID: \"4cefbe6c-67a4-4d0b-9246-29a382b4bd96\") " Oct 31 13:59:26.417645 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.417624 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-openshift-ca" pod "4cefbe6c-67a4-4d0b-9246-29a382b4bd96" (UID: "4cefbe6c-67a4-4d0b-9246-29a382b4bd96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:26.418224 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.418203 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:26.419478 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.419456 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-xks-ca" pod "4cefbe6c-67a4-4d0b-9246-29a382b4bd96" (UID: "4cefbe6c-67a4-4d0b-9246-29a382b4bd96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:59:26.419549 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.419503 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-kube-api-access-d929k" pod "4cefbe6c-67a4-4d0b-9246-29a382b4bd96" (UID: "4cefbe6c-67a4-4d0b-9246-29a382b4bd96"). InnerVolumeSpecName "kube-api-access-d929k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:26.491316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.491269 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv"] Oct 31 13:59:26.498151 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.498123 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-7b3ddd-75f47f5cc5-2c4hv"] Oct 31 13:59:26.508799 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.508774 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq"] Oct 31 13:59:26.512807 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.512787 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-7b3ddd-5d6544b5f5-v6tzq"] Oct 31 13:59:26.519945 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.519923 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d929k\" (UniqueName: \"kubernetes.io/projected/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-kube-api-access-d929k\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:26.520021 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:26.519946 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cefbe6c-67a4-4d0b-9246-29a382b4bd96-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:27.171432 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:27.171397 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" Oct 31 13:59:27.171432 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:27.171423 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7" event={"ID":"4cefbe6c-67a4-4d0b-9246-29a382b4bd96","Type":"ContainerDied","Data":"53296acfeca2db38dd9f6de6eb5c3f4bf678ca09776b6c7e78c4148fe2067c0c"} Oct 31 13:59:27.171976 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:27.171480 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53296acfeca2db38dd9f6de6eb5c3f4bf678ca09776b6c7e78c4148fe2067c0c" Oct 31 13:59:28.010199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:28.010159 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93db7894-3327-4e51-8c0a-81a77124dd7a" path="/var/lib/kubelet/pods/93db7894-3327-4e51-8c0a-81a77124dd7a/volumes" Oct 31 13:59:28.010838 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:28.010820 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb7b65f-cd03-4643-9a7a-e246ddeafe52" path="/var/lib/kubelet/pods/dcb7b65f-cd03-4643-9a7a-e246ddeafe52/volumes" Oct 31 13:59:31.763683 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:31.763585 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 13:59:37.636991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:37.636333 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz"] Oct 31 13:59:37.636991 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:37.636651 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" podUID="93d476d7-a18f-4ac0-a11d-3fb38b57d429" containerName="payload-processing" containerID="cri-o://7adb0c9dbacd4095c97cbcb0aa76e8fe122a0b844564d22decb0818ff546c2b6" gracePeriod=30 Oct 31 13:59:37.650868 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:37.650836 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6"] Oct 31 13:59:37.653830 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:37.653775 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" podUID="d7a200fc-140b-432d-8bf2-47f7a892bca7" containerName="payload-pre-processing" containerID="cri-o://047ec799462ee77cef1d5e4eb6956933ff93e35b105d22853cbacfb9f746738f" gracePeriod=30 Oct 31 13:59:38.105846 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.105800 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" Oct 31 13:59:38.172922 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.172887 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6"] Oct 31 13:59:38.178702 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.178678 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e50b22-qg5s6"] Oct 31 13:59:38.201400 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.201376 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" Oct 31 13:59:38.206771 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.206748 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d476d7-a18f-4ac0-a11d-3fb38b57d429-kube-api-access-c7lhf\" (UniqueName: \"kubernetes.io/projected/93d476d7-a18f-4ac0-a11d-3fb38b57d429-kube-api-access-c7lhf\") pod \"93d476d7-a18f-4ac0-a11d-3fb38b57d429\" (UID: \"93d476d7-a18f-4ac0-a11d-3fb38b57d429\") " Oct 31 13:59:38.206896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.206801 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7a200fc-140b-432d-8bf2-47f7a892bca7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a200fc-140b-432d-8bf2-47f7a892bca7-plugins-config-volume\") pod \"d7a200fc-140b-432d-8bf2-47f7a892bca7\" (UID: \"d7a200fc-140b-432d-8bf2-47f7a892bca7\") " Oct 31 13:59:38.206896 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.206855 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d476d7-a18f-4ac0-a11d-3fb38b57d429-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d476d7-a18f-4ac0-a11d-3fb38b57d429-plugins-config-volume\") pod \"93d476d7-a18f-4ac0-a11d-3fb38b57d429\" (UID: \"93d476d7-a18f-4ac0-a11d-3fb38b57d429\") " Oct 31 13:59:38.207198 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.207176 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7a200fc-140b-432d-8bf2-47f7a892bca7-kube-api-access-kqzjm\" (UniqueName: \"kubernetes.io/projected/d7a200fc-140b-432d-8bf2-47f7a892bca7-kube-api-access-kqzjm\") pod \"d7a200fc-140b-432d-8bf2-47f7a892bca7\" (UID: \"d7a200fc-140b-432d-8bf2-47f7a892bca7\") " Oct 31 13:59:38.207279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.207201 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7a200fc-140b-432d-8bf2-47f7a892bca7-plugins-config-volume" pod "d7a200fc-140b-432d-8bf2-47f7a892bca7" (UID: "d7a200fc-140b-432d-8bf2-47f7a892bca7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:38.207279 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.207213 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d476d7-a18f-4ac0-a11d-3fb38b57d429-plugins-config-volume" pod "93d476d7-a18f-4ac0-a11d-3fb38b57d429" (UID: "93d476d7-a18f-4ac0-a11d-3fb38b57d429"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:38.207875 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.207851 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7a200fc-140b-432d-8bf2-47f7a892bca7-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:38.207953 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.207877 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93d476d7-a18f-4ac0-a11d-3fb38b57d429-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:38.209162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.209139 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d476d7-a18f-4ac0-a11d-3fb38b57d429-kube-api-access-c7lhf" pod "93d476d7-a18f-4ac0-a11d-3fb38b57d429" (UID: "93d476d7-a18f-4ac0-a11d-3fb38b57d429"). InnerVolumeSpecName "kube-api-access-c7lhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:38.209162 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.209157 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7a200fc-140b-432d-8bf2-47f7a892bca7-kube-api-access-kqzjm" pod "d7a200fc-140b-432d-8bf2-47f7a892bca7" (UID: "d7a200fc-140b-432d-8bf2-47f7a892bca7"). InnerVolumeSpecName "kube-api-access-kqzjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:38.282044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.282006 5532 generic.go:356] "Generic (PLEG): container finished" podID="d7a200fc-140b-432d-8bf2-47f7a892bca7" containerID="047ec799462ee77cef1d5e4eb6956933ff93e35b105d22853cbacfb9f746738f" exitCode=0 Oct 31 13:59:38.282235 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.282057 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" Oct 31 13:59:38.283688 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.283654 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" event={"ID":"d7a200fc-140b-432d-8bf2-47f7a892bca7","Type":"ContainerDied","Data":"047ec799462ee77cef1d5e4eb6956933ff93e35b105d22853cbacfb9f746738f"} Oct 31 13:59:38.283843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.283733 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6" event={"ID":"d7a200fc-140b-432d-8bf2-47f7a892bca7","Type":"ContainerDied","Data":"1b254c2273cc3871edcc4280b93c45a5e24c65e96b9dd10f38ccb8a7c819bcad"} Oct 31 13:59:38.283843 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.283768 5532 scope.go:122] "RemoveContainer" containerID="047ec799462ee77cef1d5e4eb6956933ff93e35b105d22853cbacfb9f746738f" Oct 31 13:59:38.284587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.284562 5532 generic.go:356] "Generic (PLEG): container finished" podID="93d476d7-a18f-4ac0-a11d-3fb38b57d429" containerID="7adb0c9dbacd4095c97cbcb0aa76e8fe122a0b844564d22decb0818ff546c2b6" exitCode=0 Oct 31 13:59:38.284587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.284643 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" event={"ID":"93d476d7-a18f-4ac0-a11d-3fb38b57d429","Type":"ContainerDied","Data":"7adb0c9dbacd4095c97cbcb0aa76e8fe122a0b844564d22decb0818ff546c2b6"} Oct 31 13:59:38.284587 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.284651 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" Oct 31 13:59:38.284864 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.284688 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz" event={"ID":"93d476d7-a18f-4ac0-a11d-3fb38b57d429","Type":"ContainerDied","Data":"eeeb6550dc81233757e7a08cbe29fe128f56293793aa9fc06834c8c97c35c48b"} Oct 31 13:59:38.323282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.314057 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqzjm\" (UniqueName: \"kubernetes.io/projected/d7a200fc-140b-432d-8bf2-47f7a892bca7-kube-api-access-kqzjm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:38.323282 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.314093 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7lhf\" (UniqueName: \"kubernetes.io/projected/93d476d7-a18f-4ac0-a11d-3fb38b57d429-kube-api-access-c7lhf\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:38.338434 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.338402 5532 scope.go:122] "RemoveContainer" containerID="047ec799462ee77cef1d5e4eb6956933ff93e35b105d22853cbacfb9f746738f" Oct 31 13:59:38.338815 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:38.338793 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"047ec799462ee77cef1d5e4eb6956933ff93e35b105d22853cbacfb9f746738f\": container with ID starting with 047ec799462ee77cef1d5e4eb6956933ff93e35b105d22853cbacfb9f746738f not found: ID does not exist" containerID="047ec799462ee77cef1d5e4eb6956933ff93e35b105d22853cbacfb9f746738f" Oct 31 13:59:38.338903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.338823 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"047ec799462ee77cef1d5e4eb6956933ff93e35b105d22853cbacfb9f746738f"} err="failed to get container status \"047ec799462ee77cef1d5e4eb6956933ff93e35b105d22853cbacfb9f746738f\": rpc error: code = NotFound desc = could not find container \"047ec799462ee77cef1d5e4eb6956933ff93e35b105d22853cbacfb9f746738f\": container with ID starting with 047ec799462ee77cef1d5e4eb6956933ff93e35b105d22853cbacfb9f746738f not found: ID does not exist" Oct 31 13:59:38.338903 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.338843 5532 scope.go:122] "RemoveContainer" containerID="7adb0c9dbacd4095c97cbcb0aa76e8fe122a0b844564d22decb0818ff546c2b6" Oct 31 13:59:38.349779 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.349754 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6"] Oct 31 13:59:38.353088 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.353069 5532 scope.go:122] "RemoveContainer" containerID="7adb0c9dbacd4095c97cbcb0aa76e8fe122a0b844564d22decb0818ff546c2b6" Oct 31 13:59:38.353406 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:38.353385 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7adb0c9dbacd4095c97cbcb0aa76e8fe122a0b844564d22decb0818ff546c2b6\": container with ID starting with 7adb0c9dbacd4095c97cbcb0aa76e8fe122a0b844564d22decb0818ff546c2b6 not found: ID does not exist" containerID="7adb0c9dbacd4095c97cbcb0aa76e8fe122a0b844564d22decb0818ff546c2b6" Oct 31 13:59:38.353484 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.353414 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7adb0c9dbacd4095c97cbcb0aa76e8fe122a0b844564d22decb0818ff546c2b6"} err="failed to get container status \"7adb0c9dbacd4095c97cbcb0aa76e8fe122a0b844564d22decb0818ff546c2b6\": rpc error: code = NotFound desc = could not find container \"7adb0c9dbacd4095c97cbcb0aa76e8fe122a0b844564d22decb0818ff546c2b6\": container with ID starting with 7adb0c9dbacd4095c97cbcb0aa76e8fe122a0b844564d22decb0818ff546c2b6 not found: ID does not exist" Oct 31 13:59:38.354103 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.354081 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e50b22-6dc57f5487-hfgl6"] Oct 31 13:59:38.364028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.363997 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz"] Oct 31 13:59:38.367899 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:38.367878 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e50b22-7787cd6b66-xfgwz"] Oct 31 13:59:40.007403 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:40.007371 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d476d7-a18f-4ac0-a11d-3fb38b57d429" path="/var/lib/kubelet/pods/93d476d7-a18f-4ac0-a11d-3fb38b57d429/volumes" Oct 31 13:59:40.007913 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:40.007899 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="971eccfa-4d3c-4154-9870-388b6ca5e9f8" path="/var/lib/kubelet/pods/971eccfa-4d3c-4154-9870-388b6ca5e9f8/volumes" Oct 31 13:59:40.008289 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:40.008276 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7a200fc-140b-432d-8bf2-47f7a892bca7" path="/var/lib/kubelet/pods/d7a200fc-140b-432d-8bf2-47f7a892bca7/volumes" Oct 31 13:59:40.639895 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:40.639863 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6"] Oct 31 13:59:40.646296 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:40.646269 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-7b3ddd-s7tj6"] Oct 31 13:59:42.030309 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:42.030269 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25ae9bf8-0079-47e4-ab79-048d3ec452a9" path="/var/lib/kubelet/pods/25ae9bf8-0079-47e4-ab79-048d3ec452a9/volumes" Oct 31 13:59:46.414664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.409867 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz"] Oct 31 13:59:46.414664 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.410084 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" containerID="cri-o://8119b1408365d460aaf0400d0aa98647485fa81ed1fb7494280e7df5bf7bcc4e" gracePeriod=30 Oct 31 13:59:46.912924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.912894 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr"] Oct 31 13:59:46.913617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913586 5532 cpu_manager.go:414] "containerMap: removing container" podUID="dcb7b65f-cd03-4643-9a7a-e246ddeafe52" containerName="payload-processing" Oct 31 13:59:46.913669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913619 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb7b65f-cd03-4643-9a7a-e246ddeafe52" containerName="payload-processing" Oct 31 13:59:46.913669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913635 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d7a200fc-140b-432d-8bf2-47f7a892bca7" containerName="payload-pre-processing" Oct 31 13:59:46.913669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913640 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7a200fc-140b-432d-8bf2-47f7a892bca7" containerName="payload-pre-processing" Oct 31 13:59:46.913669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913656 5532 cpu_manager.go:414] "containerMap: removing container" podUID="93db7894-3327-4e51-8c0a-81a77124dd7a" containerName="payload-pre-processing" Oct 31 13:59:46.913669 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913661 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93db7894-3327-4e51-8c0a-81a77124dd7a" containerName="payload-pre-processing" Oct 31 13:59:46.913839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913681 5532 cpu_manager.go:414] "containerMap: removing container" podUID="93d476d7-a18f-4ac0-a11d-3fb38b57d429" containerName="payload-processing" Oct 31 13:59:46.913839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913685 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d476d7-a18f-4ac0-a11d-3fb38b57d429" containerName="payload-processing" Oct 31 13:59:46.913839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913721 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4cefbe6c-67a4-4d0b-9246-29a382b4bd96" containerName="revoke-keys" Oct 31 13:59:46.913839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913726 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cefbe6c-67a4-4d0b-9246-29a382b4bd96" containerName="revoke-keys" Oct 31 13:59:46.913839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913828 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb7b65f-cd03-4643-9a7a-e246ddeafe52" containerName="payload-processing" Oct 31 13:59:46.913839 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913839 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7a200fc-140b-432d-8bf2-47f7a892bca7" containerName="payload-pre-processing" Oct 31 13:59:46.914028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913846 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="93db7894-3327-4e51-8c0a-81a77124dd7a" containerName="payload-pre-processing" Oct 31 13:59:46.914028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913855 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d476d7-a18f-4ac0-a11d-3fb38b57d429" containerName="payload-processing" Oct 31 13:59:46.914028 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.913866 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cefbe6c-67a4-4d0b-9246-29a382b4bd96" containerName="revoke-keys" Oct 31 13:59:46.919464 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.919445 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" Oct 31 13:59:46.923964 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.923924 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:59:46.924462 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.923984 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:59:46.927059 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:46.927021 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr"] Oct 31 13:59:47.089105 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:47.089072 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l8wk\" (UniqueName: \"kubernetes.io/projected/5d79fc86-7404-4e6f-bd47-14c06efd584d-kube-api-access-5l8wk\") pod \"maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr\" (UID: \"5d79fc86-7404-4e6f-bd47-14c06efd584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" Oct 31 13:59:47.089300 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:47.089208 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr\" (UID: \"5d79fc86-7404-4e6f-bd47-14c06efd584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" Oct 31 13:59:47.089534 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:47.089505 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr\" (UID: \"5d79fc86-7404-4e6f-bd47-14c06efd584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" Oct 31 13:59:47.192042 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:47.191951 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr\" (UID: \"5d79fc86-7404-4e6f-bd47-14c06efd584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" Oct 31 13:59:47.192254 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:47.192236 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l8wk\" (UniqueName: \"kubernetes.io/projected/5d79fc86-7404-4e6f-bd47-14c06efd584d-kube-api-access-5l8wk\") pod \"maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr\" (UID: \"5d79fc86-7404-4e6f-bd47-14c06efd584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" Oct 31 13:59:47.192337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:47.192322 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr\" (UID: \"5d79fc86-7404-4e6f-bd47-14c06efd584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" Oct 31 13:59:47.192677 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:47.192654 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr\" (UID: \"5d79fc86-7404-4e6f-bd47-14c06efd584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" Oct 31 13:59:47.194704 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:47.194680 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr\" (UID: \"5d79fc86-7404-4e6f-bd47-14c06efd584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" Oct 31 13:59:47.205749 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:47.205716 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l8wk\" (UniqueName: \"kubernetes.io/projected/5d79fc86-7404-4e6f-bd47-14c06efd584d-kube-api-access-5l8wk\") pod \"maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr\" (UID: \"5d79fc86-7404-4e6f-bd47-14c06efd584d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" Oct 31 13:59:47.237652 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:47.237614 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" Oct 31 13:59:47.528180 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:47.528141 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr"] Oct 31 13:59:47.529456 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:59:47.529425 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d79fc86_7404_4e6f_bd47_14c06efd584d.slice/crio-3f376b1b63a24113206f65361911e36cad14d2c470a12b5c5c0fd23655b2f94e WatchSource:0}: Error finding container 3f376b1b63a24113206f65361911e36cad14d2c470a12b5c5c0fd23655b2f94e: Status 404 returned error can't find the container with id 3f376b1b63a24113206f65361911e36cad14d2c470a12b5c5c0fd23655b2f94e Oct 31 13:59:48.225675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:48.222855 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:59:48.225675 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:48.222911 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:59:48.421975 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:48.421936 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" event={"ID":"5d79fc86-7404-4e6f-bd47-14c06efd584d","Type":"ContainerStarted","Data":"ef165bec02e4019d523463ef009c1a683a7f3e35542aaa22d926721b58bf855b"} Oct 31 13:59:48.422147 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:48.421993 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" event={"ID":"5d79fc86-7404-4e6f-bd47-14c06efd584d","Type":"ContainerStarted","Data":"3f376b1b63a24113206f65361911e36cad14d2c470a12b5c5c0fd23655b2f94e"} Oct 31 13:59:48.478044 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:48.477223 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" podStartSLOduration=2.477202152 podStartE2EDuration="2.477202152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:59:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:59:48.471545813 +0000 UTC m=+3356.617289643" watchObservedRunningTime="2026-10-31 13:59:48.477202152 +0000 UTC m=+3356.622945979" Oct 31 13:59:48.682853 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:48.682814 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj"] Oct 31 13:59:48.683348 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:48.683073 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" containerID="cri-o://6b787d58998e63da4ce9230b0a58159346851e8dd2b2023522681771d04dd75a" gracePeriod=30 Oct 31 13:59:49.345670 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:49.345634 5532 patch_prober.go:29] interesting pod/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 13:59:49.345844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:49.345684 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 13:59:49.432662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:49.432567 5532 generic.go:356] "Generic (PLEG): container finished" podID="5d79fc86-7404-4e6f-bd47-14c06efd584d" containerID="ef165bec02e4019d523463ef009c1a683a7f3e35542aaa22d926721b58bf855b" exitCode=0 Oct 31 13:59:49.432662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:49.432625 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" event={"ID":"5d79fc86-7404-4e6f-bd47-14c06efd584d","Type":"ContainerDied","Data":"ef165bec02e4019d523463ef009c1a683a7f3e35542aaa22d926721b58bf855b"} Oct 31 13:59:50.655199 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:50.655170 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" Oct 31 13:59:50.686414 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:50.686379 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-xks-ca\") pod \"5d79fc86-7404-4e6f-bd47-14c06efd584d\" (UID: \"5d79fc86-7404-4e6f-bd47-14c06efd584d\") " Oct 31 13:59:50.686613 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:50.686467 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d79fc86-7404-4e6f-bd47-14c06efd584d-kube-api-access-5l8wk\" (UniqueName: \"kubernetes.io/projected/5d79fc86-7404-4e6f-bd47-14c06efd584d-kube-api-access-5l8wk\") pod \"5d79fc86-7404-4e6f-bd47-14c06efd584d\" (UID: \"5d79fc86-7404-4e6f-bd47-14c06efd584d\") " Oct 31 13:59:50.686794 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:50.686771 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-openshift-ca\") pod \"5d79fc86-7404-4e6f-bd47-14c06efd584d\" (UID: \"5d79fc86-7404-4e6f-bd47-14c06efd584d\") " Oct 31 13:59:50.687164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:50.687136 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-openshift-ca" pod "5d79fc86-7404-4e6f-bd47-14c06efd584d" (UID: "5d79fc86-7404-4e6f-bd47-14c06efd584d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:50.687383 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:50.687365 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:50.689529 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:50.689500 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-xks-ca" pod "5d79fc86-7404-4e6f-bd47-14c06efd584d" (UID: "5d79fc86-7404-4e6f-bd47-14c06efd584d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:59:50.690785 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:50.690757 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d79fc86-7404-4e6f-bd47-14c06efd584d-kube-api-access-5l8wk" pod "5d79fc86-7404-4e6f-bd47-14c06efd584d" (UID: "5d79fc86-7404-4e6f-bd47-14c06efd584d"). InnerVolumeSpecName "kube-api-access-5l8wk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:50.789115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:50.789081 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d79fc86-7404-4e6f-bd47-14c06efd584d-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:50.789115 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:50.789114 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5l8wk\" (UniqueName: \"kubernetes.io/projected/5d79fc86-7404-4e6f-bd47-14c06efd584d-kube-api-access-5l8wk\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:51.451396 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.451367 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" Oct 31 13:59:51.451518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.451392 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr" event={"ID":"5d79fc86-7404-4e6f-bd47-14c06efd584d","Type":"ContainerDied","Data":"3f376b1b63a24113206f65361911e36cad14d2c470a12b5c5c0fd23655b2f94e"} Oct 31 13:59:51.451518 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.451439 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f376b1b63a24113206f65361911e36cad14d2c470a12b5c5c0fd23655b2f94e" Oct 31 13:59:51.749811 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.749783 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:59:51.802787 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.802735 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-podinfo\") pod \"94c0b065-03aa-4039-beb9-1fcdc146296b\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " Oct 31 13:59:51.803006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.802799 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-token\" (UniqueName: \"kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-token\") pod \"94c0b065-03aa-4039-beb9-1fcdc146296b\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " Oct 31 13:59:51.803006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.802861 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-certs\") pod \"94c0b065-03aa-4039-beb9-1fcdc146296b\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " Oct 31 13:59:51.803006 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.802894 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-kube-api-access-4w824\" (UniqueName: \"kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-kube-api-access-4w824\") pod \"94c0b065-03aa-4039-beb9-1fcdc146296b\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " Oct 31 13:59:51.803187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.803087 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-credential-socket\") pod \"94c0b065-03aa-4039-beb9-1fcdc146296b\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " Oct 31 13:59:51.803187 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.803133 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c0b065-03aa-4039-beb9-1fcdc146296b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c0b065-03aa-4039-beb9-1fcdc146296b-istiod-ca-cert\") pod \"94c0b065-03aa-4039-beb9-1fcdc146296b\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " Oct 31 13:59:51.803363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.803195 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-certs" pod "94c0b065-03aa-4039-beb9-1fcdc146296b" (UID: "94c0b065-03aa-4039-beb9-1fcdc146296b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:51.803363 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.803209 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-socket\") pod \"94c0b065-03aa-4039-beb9-1fcdc146296b\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " Oct 31 13:59:51.803487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.803388 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-data\") pod \"94c0b065-03aa-4039-beb9-1fcdc146296b\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " Oct 31 13:59:51.803487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.803427 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-socket" pod "94c0b065-03aa-4039-beb9-1fcdc146296b" (UID: "94c0b065-03aa-4039-beb9-1fcdc146296b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:51.803487 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.803468 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-envoy\") pod \"94c0b065-03aa-4039-beb9-1fcdc146296b\" (UID: \"94c0b065-03aa-4039-beb9-1fcdc146296b\") " Oct 31 13:59:51.803712 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.803686 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-credential-socket" pod "94c0b065-03aa-4039-beb9-1fcdc146296b" (UID: "94c0b065-03aa-4039-beb9-1fcdc146296b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:51.803835 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.803812 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-data" pod "94c0b065-03aa-4039-beb9-1fcdc146296b" (UID: "94c0b065-03aa-4039-beb9-1fcdc146296b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:51.804038 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.804017 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c0b065-03aa-4039-beb9-1fcdc146296b-istiod-ca-cert" pod "94c0b065-03aa-4039-beb9-1fcdc146296b" (UID: "94c0b065-03aa-4039-beb9-1fcdc146296b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:51.804341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.804302 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:51.804341 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.804328 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:51.804504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.804344 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:51.804504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.804359 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c0b065-03aa-4039-beb9-1fcdc146296b-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:51.804504 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.804372 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:51.805752 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.805721 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-token" pod "94c0b065-03aa-4039-beb9-1fcdc146296b" (UID: "94c0b065-03aa-4039-beb9-1fcdc146296b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:51.807227 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.807195 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-envoy" pod "94c0b065-03aa-4039-beb9-1fcdc146296b" (UID: "94c0b065-03aa-4039-beb9-1fcdc146296b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:51.807426 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.807403 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-kube-api-access-4w824" pod "94c0b065-03aa-4039-beb9-1fcdc146296b" (UID: "94c0b065-03aa-4039-beb9-1fcdc146296b"). InnerVolumeSpecName "kube-api-access-4w824". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:51.807515 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.807492 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-podinfo" pod "94c0b065-03aa-4039-beb9-1fcdc146296b" (UID: "94c0b065-03aa-4039-beb9-1fcdc146296b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:59:51.906062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.906027 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:51.906062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.906056 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:51.906062 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.906067 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w824\" (UniqueName: \"kubernetes.io/projected/94c0b065-03aa-4039-beb9-1fcdc146296b-kube-api-access-4w824\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:51.906322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:51.906082 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c0b065-03aa-4039-beb9-1fcdc146296b-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:52.169928 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.169897 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs"] Oct 31 13:59:52.170656 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.170638 5532 cpu_manager.go:414] "containerMap: removing container" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" Oct 31 13:59:52.170716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.170655 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" Oct 31 13:59:52.170716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.170670 5532 cpu_manager.go:414] "containerMap: removing container" podUID="5d79fc86-7404-4e6f-bd47-14c06efd584d" containerName="revoke-keys" Oct 31 13:59:52.170716 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.170676 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d79fc86-7404-4e6f-bd47-14c06efd584d" containerName="revoke-keys" Oct 31 13:59:52.170847 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.170837 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" Oct 31 13:59:52.170881 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.170855 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d79fc86-7404-4e6f-bd47-14c06efd584d" containerName="revoke-keys" Oct 31 13:59:52.174947 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.174929 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" Oct 31 13:59:52.182611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.181214 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 13:59:52.182611 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.181279 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 13:59:52.185337 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.185307 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs"] Oct 31 13:59:52.209806 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.209774 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33de6359-mk6fs\" (UID: \"01ce9637-0da3-4e30-af28-2966c675990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" Oct 31 13:59:52.210014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.209849 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33de6359-mk6fs\" (UID: \"01ce9637-0da3-4e30-af28-2966c675990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" Oct 31 13:59:52.210014 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.209989 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ddj8\" (UniqueName: \"kubernetes.io/projected/01ce9637-0da3-4e30-af28-2966c675990f-kube-api-access-8ddj8\") pod \"maas-api-revoke-sub-33de6359-mk6fs\" (UID: \"01ce9637-0da3-4e30-af28-2966c675990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" Oct 31 13:59:52.311927 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.311837 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ddj8\" (UniqueName: \"kubernetes.io/projected/01ce9637-0da3-4e30-af28-2966c675990f-kube-api-access-8ddj8\") pod \"maas-api-revoke-sub-33de6359-mk6fs\" (UID: \"01ce9637-0da3-4e30-af28-2966c675990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" Oct 31 13:59:52.312155 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.312136 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33de6359-mk6fs\" (UID: \"01ce9637-0da3-4e30-af28-2966c675990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" Oct 31 13:59:52.312244 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.312227 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33de6359-mk6fs\" (UID: \"01ce9637-0da3-4e30-af28-2966c675990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" Oct 31 13:59:52.312949 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.312921 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33de6359-mk6fs\" (UID: \"01ce9637-0da3-4e30-af28-2966c675990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" Oct 31 13:59:52.314530 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.314509 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33de6359-mk6fs\" (UID: \"01ce9637-0da3-4e30-af28-2966c675990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" Oct 31 13:59:52.326002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.325976 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ddj8\" (UniqueName: \"kubernetes.io/projected/01ce9637-0da3-4e30-af28-2966c675990f-kube-api-access-8ddj8\") pod \"maas-api-revoke-sub-33de6359-mk6fs\" (UID: \"01ce9637-0da3-4e30-af28-2966c675990f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" Oct 31 13:59:52.462126 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.462098 5532 generic.go:356] "Generic (PLEG): container finished" podID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerID="8119b1408365d460aaf0400d0aa98647485fa81ed1fb7494280e7df5bf7bcc4e" exitCode=0 Oct 31 13:59:52.462316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.462145 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" Oct 31 13:59:52.462316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.462180 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" event={"ID":"94c0b065-03aa-4039-beb9-1fcdc146296b","Type":"ContainerDied","Data":"8119b1408365d460aaf0400d0aa98647485fa81ed1fb7494280e7df5bf7bcc4e"} Oct 31 13:59:52.462316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.462234 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz" event={"ID":"94c0b065-03aa-4039-beb9-1fcdc146296b","Type":"ContainerDied","Data":"a54cf1de8f932b28e64f1e963c7a1d684e2b885a0cc5d2162bfedd5bee616272"} Oct 31 13:59:52.462316 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.462257 5532 scope.go:122] "RemoveContainer" containerID="8119b1408365d460aaf0400d0aa98647485fa81ed1fb7494280e7df5bf7bcc4e" Oct 31 13:59:52.474113 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.474094 5532 scope.go:122] "RemoveContainer" containerID="7d48071c015ef2402428926132d720aaa6d1103f71dd7ae22840029cfb80091c" Oct 31 13:59:52.502844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.486256 5532 scope.go:122] "RemoveContainer" containerID="8119b1408365d460aaf0400d0aa98647485fa81ed1fb7494280e7df5bf7bcc4e" Oct 31 13:59:52.502844 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:52.486574 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8119b1408365d460aaf0400d0aa98647485fa81ed1fb7494280e7df5bf7bcc4e\": container with ID starting with 8119b1408365d460aaf0400d0aa98647485fa81ed1fb7494280e7df5bf7bcc4e not found: ID does not exist" containerID="8119b1408365d460aaf0400d0aa98647485fa81ed1fb7494280e7df5bf7bcc4e" Oct 31 13:59:52.502844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.486671 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8119b1408365d460aaf0400d0aa98647485fa81ed1fb7494280e7df5bf7bcc4e"} err="failed to get container status \"8119b1408365d460aaf0400d0aa98647485fa81ed1fb7494280e7df5bf7bcc4e\": rpc error: code = NotFound desc = could not find container \"8119b1408365d460aaf0400d0aa98647485fa81ed1fb7494280e7df5bf7bcc4e\": container with ID starting with 8119b1408365d460aaf0400d0aa98647485fa81ed1fb7494280e7df5bf7bcc4e not found: ID does not exist" Oct 31 13:59:52.502844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.486699 5532 scope.go:122] "RemoveContainer" containerID="7d48071c015ef2402428926132d720aaa6d1103f71dd7ae22840029cfb80091c" Oct 31 13:59:52.502844 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:52.486979 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d48071c015ef2402428926132d720aaa6d1103f71dd7ae22840029cfb80091c\": container with ID starting with 7d48071c015ef2402428926132d720aaa6d1103f71dd7ae22840029cfb80091c not found: ID does not exist" containerID="7d48071c015ef2402428926132d720aaa6d1103f71dd7ae22840029cfb80091c" Oct 31 13:59:52.502844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.487006 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d48071c015ef2402428926132d720aaa6d1103f71dd7ae22840029cfb80091c"} err="failed to get container status \"7d48071c015ef2402428926132d720aaa6d1103f71dd7ae22840029cfb80091c\": rpc error: code = NotFound desc = could not find container \"7d48071c015ef2402428926132d720aaa6d1103f71dd7ae22840029cfb80091c\": container with ID starting with 7d48071c015ef2402428926132d720aaa6d1103f71dd7ae22840029cfb80091c not found: ID does not exist" Oct 31 13:59:52.502844 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.501806 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" Oct 31 13:59:52.519646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.519438 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz"] Oct 31 13:59:52.525184 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.525153 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e50b22-openshift-default-ccc6dd79f-wz9lz"] Oct 31 13:59:52.806955 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:52.806920 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs"] Oct 31 13:59:52.807393 ip-10-0-128-41 kubenswrapper[5532]: W1031 13:59:52.807289 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01ce9637_0da3_4e30_af28_2966c675990f.slice/crio-0c30729b1c399e79591fec44ad2ed6004ce8aaef52b3e902d3e77c5c6a7f5928 WatchSource:0}: Error finding container 0c30729b1c399e79591fec44ad2ed6004ce8aaef52b3e902d3e77c5c6a7f5928: Status 404 returned error can't find the container with id 0c30729b1c399e79591fec44ad2ed6004ce8aaef52b3e902d3e77c5c6a7f5928 Oct 31 13:59:53.476993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:53.476916 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" event={"ID":"01ce9637-0da3-4e30-af28-2966c675990f","Type":"ContainerStarted","Data":"022b0a140dd2a317409a5beda16203b02bfd9efdb9d65abc4338db1cce2c359d"} Oct 31 13:59:53.476993 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:53.476965 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" event={"ID":"01ce9637-0da3-4e30-af28-2966c675990f","Type":"ContainerStarted","Data":"0c30729b1c399e79591fec44ad2ed6004ce8aaef52b3e902d3e77c5c6a7f5928"} Oct 31 13:59:53.499857 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:53.499804 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" podStartSLOduration=1.499789463 podStartE2EDuration="1.499789463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 13:59:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 13:59:53.496266763 +0000 UTC m=+3361.642010594" watchObservedRunningTime="2026-10-31 13:59:53.499789463 +0000 UTC m=+3361.645533325" Oct 31 13:59:54.008472 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.008388 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" path="/var/lib/kubelet/pods/94c0b065-03aa-4039-beb9-1fcdc146296b/volumes" Oct 31 13:59:54.078124 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.078100 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:59:54.079924 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.079901 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 13:59:54.164620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.154374 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/928c06ae-3fb5-4d89-aa1f-0da90832da40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928c06ae-3fb5-4d89-aa1f-0da90832da40-istiod-ca-cert\") pod \"928c06ae-3fb5-4d89-aa1f-0da90832da40\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " Oct 31 13:59:54.164620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.154443 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-podinfo\") pod \"928c06ae-3fb5-4d89-aa1f-0da90832da40\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " Oct 31 13:59:54.164620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.154498 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-data\") pod \"928c06ae-3fb5-4d89-aa1f-0da90832da40\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " Oct 31 13:59:54.164620 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.154543 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-envoy\") pod \"928c06ae-3fb5-4d89-aa1f-0da90832da40\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " Oct 31 13:59:54.184772 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.184731 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-token\" (UniqueName: \"kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-token\") pod \"928c06ae-3fb5-4d89-aa1f-0da90832da40\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " Oct 31 13:59:54.185160 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.185122 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-kube-api-access-rc6kt\" (UniqueName: \"kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-kube-api-access-rc6kt\") pod \"928c06ae-3fb5-4d89-aa1f-0da90832da40\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " Oct 31 13:59:54.195181 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.195147 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-socket\") pod \"928c06ae-3fb5-4d89-aa1f-0da90832da40\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " Oct 31 13:59:54.195414 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.195398 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-credential-socket\") pod \"928c06ae-3fb5-4d89-aa1f-0da90832da40\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " Oct 31 13:59:54.197789 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.197762 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-certs\") pod \"928c06ae-3fb5-4d89-aa1f-0da90832da40\" (UID: \"928c06ae-3fb5-4d89-aa1f-0da90832da40\") " Oct 31 13:59:54.200071 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.195685 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-data" pod "928c06ae-3fb5-4d89-aa1f-0da90832da40" (UID: "928c06ae-3fb5-4d89-aa1f-0da90832da40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:54.200176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.199578 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-socket" pod "928c06ae-3fb5-4d89-aa1f-0da90832da40" (UID: "928c06ae-3fb5-4d89-aa1f-0da90832da40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:54.200176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.199776 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928c06ae-3fb5-4d89-aa1f-0da90832da40-istiod-ca-cert" pod "928c06ae-3fb5-4d89-aa1f-0da90832da40" (UID: "928c06ae-3fb5-4d89-aa1f-0da90832da40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:54.200176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.199975 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-credential-socket" pod "928c06ae-3fb5-4d89-aa1f-0da90832da40" (UID: "928c06ae-3fb5-4d89-aa1f-0da90832da40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:54.200176 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.200041 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-certs" pod "928c06ae-3fb5-4d89-aa1f-0da90832da40" (UID: "928c06ae-3fb5-4d89-aa1f-0da90832da40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:54.251617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.250167 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-kube-api-access-rc6kt" pod "928c06ae-3fb5-4d89-aa1f-0da90832da40" (UID: "928c06ae-3fb5-4d89-aa1f-0da90832da40"). InnerVolumeSpecName "kube-api-access-rc6kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:54.251617 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.250351 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-envoy" pod "928c06ae-3fb5-4d89-aa1f-0da90832da40" (UID: "928c06ae-3fb5-4d89-aa1f-0da90832da40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 13:59:54.264218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.258732 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-podinfo" pod "928c06ae-3fb5-4d89-aa1f-0da90832da40" (UID: "928c06ae-3fb5-4d89-aa1f-0da90832da40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 13:59:54.264218 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.258767 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-token" pod "928c06ae-3fb5-4d89-aa1f-0da90832da40" (UID: "928c06ae-3fb5-4d89-aa1f-0da90832da40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:54.300646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.300491 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:54.300646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.300521 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/928c06ae-3fb5-4d89-aa1f-0da90832da40-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:54.300646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.300532 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:54.300646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.300542 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:54.300646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.300558 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:54.300646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.300619 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:54.300646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.300634 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc6kt\" (UniqueName: \"kubernetes.io/projected/928c06ae-3fb5-4d89-aa1f-0da90832da40-kube-api-access-rc6kt\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:54.300646 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.300648 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:54.301306 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.300671 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/928c06ae-3fb5-4d89-aa1f-0da90832da40-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:54.523242 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.522873 5532 generic.go:356] "Generic (PLEG): container finished" podID="01ce9637-0da3-4e30-af28-2966c675990f" containerID="022b0a140dd2a317409a5beda16203b02bfd9efdb9d65abc4338db1cce2c359d" exitCode=0 Oct 31 13:59:54.523523 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.523493 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" event={"ID":"01ce9637-0da3-4e30-af28-2966c675990f","Type":"ContainerDied","Data":"022b0a140dd2a317409a5beda16203b02bfd9efdb9d65abc4338db1cce2c359d"} Oct 31 13:59:54.525978 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.525948 5532 generic.go:356] "Generic (PLEG): container finished" podID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerID="6b787d58998e63da4ce9230b0a58159346851e8dd2b2023522681771d04dd75a" exitCode=0 Oct 31 13:59:54.526116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.526012 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" Oct 31 13:59:54.526116 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.526031 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" event={"ID":"928c06ae-3fb5-4d89-aa1f-0da90832da40","Type":"ContainerDied","Data":"6b787d58998e63da4ce9230b0a58159346851e8dd2b2023522681771d04dd75a"} Oct 31 13:59:54.526221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.526135 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj" event={"ID":"928c06ae-3fb5-4d89-aa1f-0da90832da40","Type":"ContainerDied","Data":"4eaf4664008e3c63eb99393b7cb6e4cbc61ceb4ce0187c2f3240cdc075643e8e"} Oct 31 13:59:54.526221 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.526146 5532 scope.go:122] "RemoveContainer" containerID="6b787d58998e63da4ce9230b0a58159346851e8dd2b2023522681771d04dd75a" Oct 31 13:59:54.540417 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.540396 5532 scope.go:122] "RemoveContainer" containerID="a6db087353b60087ed320ed226d37351b82d2bda2c61292278f84a9a6102f5b3" Oct 31 13:59:54.553159 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.553143 5532 scope.go:122] "RemoveContainer" containerID="6b787d58998e63da4ce9230b0a58159346851e8dd2b2023522681771d04dd75a" Oct 31 13:59:54.553518 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:54.553498 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b787d58998e63da4ce9230b0a58159346851e8dd2b2023522681771d04dd75a\": container with ID starting with 6b787d58998e63da4ce9230b0a58159346851e8dd2b2023522681771d04dd75a not found: ID does not exist" containerID="6b787d58998e63da4ce9230b0a58159346851e8dd2b2023522681771d04dd75a" Oct 31 13:59:54.553607 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.553525 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b787d58998e63da4ce9230b0a58159346851e8dd2b2023522681771d04dd75a"} err="failed to get container status \"6b787d58998e63da4ce9230b0a58159346851e8dd2b2023522681771d04dd75a\": rpc error: code = NotFound desc = could not find container \"6b787d58998e63da4ce9230b0a58159346851e8dd2b2023522681771d04dd75a\": container with ID starting with 6b787d58998e63da4ce9230b0a58159346851e8dd2b2023522681771d04dd75a not found: ID does not exist" Oct 31 13:59:54.553607 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.553543 5532 scope.go:122] "RemoveContainer" containerID="a6db087353b60087ed320ed226d37351b82d2bda2c61292278f84a9a6102f5b3" Oct 31 13:59:54.553869 ip-10-0-128-41 kubenswrapper[5532]: E1031 13:59:54.553850 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6db087353b60087ed320ed226d37351b82d2bda2c61292278f84a9a6102f5b3\": container with ID starting with a6db087353b60087ed320ed226d37351b82d2bda2c61292278f84a9a6102f5b3 not found: ID does not exist" containerID="a6db087353b60087ed320ed226d37351b82d2bda2c61292278f84a9a6102f5b3" Oct 31 13:59:54.553935 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.553876 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6db087353b60087ed320ed226d37351b82d2bda2c61292278f84a9a6102f5b3"} err="failed to get container status \"a6db087353b60087ed320ed226d37351b82d2bda2c61292278f84a9a6102f5b3\": rpc error: code = NotFound desc = could not find container \"a6db087353b60087ed320ed226d37351b82d2bda2c61292278f84a9a6102f5b3\": container with ID starting with a6db087353b60087ed320ed226d37351b82d2bda2c61292278f84a9a6102f5b3 not found: ID does not exist" Oct 31 13:59:54.567326 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.567299 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj"] Oct 31 13:59:54.571581 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:54.571563 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-7b3ddd-openshift-default-5844f58fbf-m4hhj"] Oct 31 13:59:55.836167 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:55.836140 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" Oct 31 13:59:55.937825 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:55.937788 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-openshift-ca\") pod \"01ce9637-0da3-4e30-af28-2966c675990f\" (UID: \"01ce9637-0da3-4e30-af28-2966c675990f\") " Oct 31 13:59:55.938025 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:55.937887 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-xks-ca\") pod \"01ce9637-0da3-4e30-af28-2966c675990f\" (UID: \"01ce9637-0da3-4e30-af28-2966c675990f\") " Oct 31 13:59:55.938098 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:55.938057 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01ce9637-0da3-4e30-af28-2966c675990f-kube-api-access-8ddj8\" (UniqueName: \"kubernetes.io/projected/01ce9637-0da3-4e30-af28-2966c675990f-kube-api-access-8ddj8\") pod \"01ce9637-0da3-4e30-af28-2966c675990f\" (UID: \"01ce9637-0da3-4e30-af28-2966c675990f\") " Oct 31 13:59:55.938164 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:55.938132 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-openshift-ca" pod "01ce9637-0da3-4e30-af28-2966c675990f" (UID: "01ce9637-0da3-4e30-af28-2966c675990f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 13:59:55.938723 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:55.938696 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:55.940662 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:55.940632 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-xks-ca" pod "01ce9637-0da3-4e30-af28-2966c675990f" (UID: "01ce9637-0da3-4e30-af28-2966c675990f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 13:59:55.940802 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:55.940756 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01ce9637-0da3-4e30-af28-2966c675990f-kube-api-access-8ddj8" pod "01ce9637-0da3-4e30-af28-2966c675990f" (UID: "01ce9637-0da3-4e30-af28-2966c675990f"). InnerVolumeSpecName "kube-api-access-8ddj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 13:59:56.008203 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:56.008172 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" path="/var/lib/kubelet/pods/928c06ae-3fb5-4d89-aa1f-0da90832da40/volumes" Oct 31 13:59:56.040322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:56.040282 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/01ce9637-0da3-4e30-af28-2966c675990f-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:56.040322 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:56.040316 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ddj8\" (UniqueName: \"kubernetes.io/projected/01ce9637-0da3-4e30-af28-2966c675990f-kube-api-access-8ddj8\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 13:59:56.199820 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:56.199786 5532 scope.go:122] "RemoveContainer" containerID="0ba6b770ae96e4759aa5cceaafae478de7e34f7bf0938463e4be39e8b8bcc573" Oct 31 13:59:56.211575 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:56.211548 5532 scope.go:122] "RemoveContainer" containerID="aafd281455ecedc32bda4bb05721832e347337c81d403c124cb5461892d26de6" Oct 31 13:59:56.222002 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:56.221975 5532 scope.go:122] "RemoveContainer" containerID="f6317c268489b26a5909ddc687d4f19ed37b79edee559072899bacbed18ad2e1" Oct 31 13:59:56.232158 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:56.232136 5532 scope.go:122] "RemoveContainer" containerID="2c114233e421efb74d2da78296ae1eb5869c46ff7b9e08c6c71b483105d4ef46" Oct 31 13:59:56.548381 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:56.548302 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" Oct 31 13:59:56.548381 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:56.548331 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs" event={"ID":"01ce9637-0da3-4e30-af28-2966c675990f","Type":"ContainerDied","Data":"0c30729b1c399e79591fec44ad2ed6004ce8aaef52b3e902d3e77c5c6a7f5928"} Oct 31 13:59:56.548381 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:56.548379 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c30729b1c399e79591fec44ad2ed6004ce8aaef52b3e902d3e77c5c6a7f5928" Oct 31 13:59:59.029706 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:59.029673 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96"] Oct 31 13:59:59.036391 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:59.036357 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb"] Oct 31 13:59:59.040775 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:59.040748 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a08c615e-c5z96"] Oct 31 13:59:59.044428 ip-10-0-128-41 kubenswrapper[5532]: I1031 13:59:59.044404 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41c4bc5b-9bqqb"] Oct 31 14:00:00.005920 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.005882 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dce842d-b481-45e1-b014-fb11bc53a30f" path="/var/lib/kubelet/pods/0dce842d-b481-45e1-b014-fb11bc53a30f/volumes" Oct 31 14:00:00.006426 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.006411 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb1fc88-f162-4f68-bd7e-522bcd20099e" path="/var/lib/kubelet/pods/5cb1fc88-f162-4f68-bd7e-522bcd20099e/volumes" Oct 31 14:00:00.143815 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.143784 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv"] Oct 31 14:00:00.144549 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.144531 5532 cpu_manager.go:414] "containerMap: removing container" podUID="01ce9637-0da3-4e30-af28-2966c675990f" containerName="revoke-keys" Oct 31 14:00:00.144549 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.144549 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01ce9637-0da3-4e30-af28-2966c675990f" containerName="revoke-keys" Oct 31 14:00:00.144745 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.144562 5532 cpu_manager.go:414] "containerMap: removing container" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" Oct 31 14:00:00.144745 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.144568 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" Oct 31 14:00:00.144745 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.144590 5532 cpu_manager.go:414] "containerMap: removing container" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" Oct 31 14:00:00.144745 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.144605 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" Oct 31 14:00:00.144745 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.144614 5532 cpu_manager.go:414] "containerMap: removing container" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" Oct 31 14:00:00.144745 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.144620 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" Oct 31 14:00:00.145040 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.144808 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c0b065-03aa-4039-beb9-1fcdc146296b" containerName="istio-proxy" Oct 31 14:00:00.145040 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.144833 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="01ce9637-0da3-4e30-af28-2966c675990f" containerName="revoke-keys" Oct 31 14:00:00.145040 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.144846 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" Oct 31 14:00:00.145040 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.144861 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="928c06ae-3fb5-4d89-aa1f-0da90832da40" containerName="istio-proxy" Oct 31 14:00:00.150932 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.150909 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8"] Oct 31 14:00:00.151060 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.151031 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" Oct 31 14:00:00.159196 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.159169 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv"] Oct 31 14:00:00.159573 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.159449 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" Oct 31 14:00:00.161814 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.161788 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:00:00.167864 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.167821 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5"] Oct 31 14:00:00.170037 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.169724 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" Oct 31 14:00:00.191876 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.191848 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv"] Oct 31 14:00:00.192195 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.192172 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8"] Oct 31 14:00:00.192307 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.192208 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv"] Oct 31 14:00:00.192307 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.192236 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr"] Oct 31 14:00:00.193582 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.192124 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" Oct 31 14:00:00.207577 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.207427 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5"] Oct 31 14:00:00.207577 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.207477 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4"] Oct 31 14:00:00.208761 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.208735 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" Oct 31 14:00:00.213578 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.212893 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" Oct 31 14:00:00.226713 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.226586 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr"] Oct 31 14:00:00.227831 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.227797 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brkgn\" (UniqueName: \"kubernetes.io/projected/4a6c45b7-8d74-481a-b429-cc7c65420bdd-kube-api-access-brkgn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8\" (UID: \"4a6c45b7-8d74-481a-b429-cc7c65420bdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" Oct 31 14:00:00.227989 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.227899 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv\" (UID: \"d5bd142e-0a36-4389-a6f7-da08f0e4c77a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" Oct 31 14:00:00.228060 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.228007 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv\" (UID: \"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" Oct 31 14:00:00.228060 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.228043 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6c45b7-8d74-481a-b429-cc7c65420bdd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8\" (UID: \"4a6c45b7-8d74-481a-b429-cc7c65420bdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" Oct 31 14:00:00.228480 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.228183 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzwx5\" (UniqueName: \"kubernetes.io/projected/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-kube-api-access-gzwx5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv\" (UID: \"d5bd142e-0a36-4389-a6f7-da08f0e4c77a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" Oct 31 14:00:00.231134 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.228298 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnlqw\" (UniqueName: \"kubernetes.io/projected/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-kube-api-access-rnlqw\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv\" (UID: \"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" Oct 31 14:00:00.249295 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.249258 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4"] Oct 31 14:00:00.275749 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.274080 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8"] Oct 31 14:00:00.281118 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.281090 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" Oct 31 14:00:00.287466 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.287433 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl"] Oct 31 14:00:00.294477 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.293630 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" Oct 31 14:00:00.294477 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.294146 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8"] Oct 31 14:00:00.298137 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.298115 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54"] Oct 31 14:00:00.304241 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.304213 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" Oct 31 14:00:00.307912 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.307877 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl"] Oct 31 14:00:00.312901 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.312880 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2"] Oct 31 14:00:00.319138 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.319114 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54"] Oct 31 14:00:00.319304 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.319211 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" Oct 31 14:00:00.323793 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.323768 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2"] Oct 31 14:00:00.332520 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.332493 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr\" (UID: \"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" Oct 31 14:00:00.332681 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.332553 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnlqw\" (UniqueName: \"kubernetes.io/projected/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-kube-api-access-rnlqw\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv\" (UID: \"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" Oct 31 14:00:00.332681 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.332573 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltlz4\" (UniqueName: \"kubernetes.io/projected/65744c55-b224-4b04-8705-b26423016c51-kube-api-access-ltlz4\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5\" (UID: \"65744c55-b224-4b04-8705-b26423016c51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" Oct 31 14:00:00.332839 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.332816 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65744c55-b224-4b04-8705-b26423016c51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5\" (UID: \"65744c55-b224-4b04-8705-b26423016c51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" Oct 31 14:00:00.332914 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.332891 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brkgn\" (UniqueName: \"kubernetes.io/projected/4a6c45b7-8d74-481a-b429-cc7c65420bdd-kube-api-access-brkgn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8\" (UID: \"4a6c45b7-8d74-481a-b429-cc7c65420bdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" Oct 31 14:00:00.332969 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.332932 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr2vv\" (UniqueName: \"kubernetes.io/projected/d5ab1aa4-e729-443c-a322-1a4ce645be61-kube-api-access-xr2vv\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4\" (UID: \"d5ab1aa4-e729-443c-a322-1a4ce645be61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" Oct 31 14:00:00.333028 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.332993 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv\" (UID: \"d5bd142e-0a36-4389-a6f7-da08f0e4c77a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" Oct 31 14:00:00.333092 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.333075 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ab1aa4-e729-443c-a322-1a4ce645be61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4\" (UID: \"d5ab1aa4-e729-443c-a322-1a4ce645be61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" Oct 31 14:00:00.333149 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.333117 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr9q2\" (UniqueName: \"kubernetes.io/projected/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-kube-api-access-wr9q2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr\" (UID: \"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" Oct 31 14:00:00.333205 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.333184 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv\" (UID: \"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" Oct 31 14:00:00.333244 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.333219 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6c45b7-8d74-481a-b429-cc7c65420bdd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8\" (UID: \"4a6c45b7-8d74-481a-b429-cc7c65420bdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" Oct 31 14:00:00.333317 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.333302 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzwx5\" (UniqueName: \"kubernetes.io/projected/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-kube-api-access-gzwx5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv\" (UID: \"d5bd142e-0a36-4389-a6f7-da08f0e4c77a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" Oct 31 14:00:00.333752 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.333730 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv\" (UID: \"d5bd142e-0a36-4389-a6f7-da08f0e4c77a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" Oct 31 14:00:00.333857 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.333841 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv\" (UID: \"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" Oct 31 14:00:00.333963 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.333946 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6c45b7-8d74-481a-b429-cc7c65420bdd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8\" (UID: \"4a6c45b7-8d74-481a-b429-cc7c65420bdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" Oct 31 14:00:00.347022 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.346997 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnlqw\" (UniqueName: \"kubernetes.io/projected/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-kube-api-access-rnlqw\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv\" (UID: \"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" Oct 31 14:00:00.347423 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.347401 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brkgn\" (UniqueName: \"kubernetes.io/projected/4a6c45b7-8d74-481a-b429-cc7c65420bdd-kube-api-access-brkgn\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8\" (UID: \"4a6c45b7-8d74-481a-b429-cc7c65420bdd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" Oct 31 14:00:00.347423 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.347416 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzwx5\" (UniqueName: \"kubernetes.io/projected/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-kube-api-access-gzwx5\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv\" (UID: \"d5bd142e-0a36-4389-a6f7-da08f0e4c77a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" Oct 31 14:00:00.434548 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.434507 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ab1aa4-e729-443c-a322-1a4ce645be61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4\" (UID: \"d5ab1aa4-e729-443c-a322-1a4ce645be61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" Oct 31 14:00:00.434798 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.434553 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr9q2\" (UniqueName: \"kubernetes.io/projected/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-kube-api-access-wr9q2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr\" (UID: \"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" Oct 31 14:00:00.434798 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.434645 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqzcf\" (UniqueName: \"kubernetes.io/projected/d2d6b4fa-53be-495d-ba97-f8edf753caa4-kube-api-access-nqzcf\") pod \"maas-api-key-cleanup-29890920-f5wbl\" (UID: \"d2d6b4fa-53be-495d-ba97-f8edf753caa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" Oct 31 14:00:00.434798 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.434674 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641d472a-fcc8-4c9c-bac0-a721550149a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54\" (UID: \"641d472a-fcc8-4c9c-bac0-a721550149a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" Oct 31 14:00:00.434939 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.434859 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m2p7\" (UniqueName: \"kubernetes.io/projected/27c88326-5db1-4240-bbb5-75b2ed7a72a9-kube-api-access-6m2p7\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8\" (UID: \"27c88326-5db1-4240-bbb5-75b2ed7a72a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" Oct 31 14:00:00.434939 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.434925 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr\" (UID: \"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" Oct 31 14:00:00.435052 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.434947 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x9j7\" (UniqueName: \"kubernetes.io/projected/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-kube-api-access-2x9j7\") pod \"maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2\" (UID: \"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" Oct 31 14:00:00.435052 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.434996 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltlz4\" (UniqueName: \"kubernetes.io/projected/65744c55-b224-4b04-8705-b26423016c51-kube-api-access-ltlz4\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5\" (UID: \"65744c55-b224-4b04-8705-b26423016c51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" Oct 31 14:00:00.435172 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.435105 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c88326-5db1-4240-bbb5-75b2ed7a72a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8\" (UID: \"27c88326-5db1-4240-bbb5-75b2ed7a72a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" Oct 31 14:00:00.435172 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.435137 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65744c55-b224-4b04-8705-b26423016c51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5\" (UID: \"65744c55-b224-4b04-8705-b26423016c51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" Oct 31 14:00:00.435295 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.435201 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ab1aa4-e729-443c-a322-1a4ce645be61-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4\" (UID: \"d5ab1aa4-e729-443c-a322-1a4ce645be61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" Oct 31 14:00:00.435295 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.435232 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr2vv\" (UniqueName: \"kubernetes.io/projected/d5ab1aa4-e729-443c-a322-1a4ce645be61-kube-api-access-xr2vv\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4\" (UID: \"d5ab1aa4-e729-443c-a322-1a4ce645be61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" Oct 31 14:00:00.435414 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.435310 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2\" (UID: \"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" Oct 31 14:00:00.435414 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.435372 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d6b4fa-53be-495d-ba97-f8edf753caa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890920-f5wbl\" (UID: \"d2d6b4fa-53be-495d-ba97-f8edf753caa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" Oct 31 14:00:00.435529 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.435427 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w5mq\" (UniqueName: \"kubernetes.io/projected/641d472a-fcc8-4c9c-bac0-a721550149a0-kube-api-access-4w5mq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54\" (UID: \"641d472a-fcc8-4c9c-bac0-a721550149a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" Oct 31 14:00:00.435728 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.435702 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr\" (UID: \"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" Oct 31 14:00:00.435812 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.435774 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65744c55-b224-4b04-8705-b26423016c51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5\" (UID: \"65744c55-b224-4b04-8705-b26423016c51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" Oct 31 14:00:00.448522 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.448491 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr2vv\" (UniqueName: \"kubernetes.io/projected/d5ab1aa4-e729-443c-a322-1a4ce645be61-kube-api-access-xr2vv\") pod \"maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4\" (UID: \"d5ab1aa4-e729-443c-a322-1a4ce645be61\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" Oct 31 14:00:00.448522 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.448517 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr9q2\" (UniqueName: \"kubernetes.io/projected/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-kube-api-access-wr9q2\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr\" (UID: \"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" Oct 31 14:00:00.448962 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.448938 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltlz4\" (UniqueName: \"kubernetes.io/projected/65744c55-b224-4b04-8705-b26423016c51-kube-api-access-ltlz4\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5\" (UID: \"65744c55-b224-4b04-8705-b26423016c51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" Oct 31 14:00:00.490957 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.490908 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" Oct 31 14:00:00.511857 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.511830 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" Oct 31 14:00:00.526413 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.526388 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" Oct 31 14:00:00.538874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.536993 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqzcf\" (UniqueName: \"kubernetes.io/projected/d2d6b4fa-53be-495d-ba97-f8edf753caa4-kube-api-access-nqzcf\") pod \"maas-api-key-cleanup-29890920-f5wbl\" (UID: \"d2d6b4fa-53be-495d-ba97-f8edf753caa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" Oct 31 14:00:00.538874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.537041 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641d472a-fcc8-4c9c-bac0-a721550149a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54\" (UID: \"641d472a-fcc8-4c9c-bac0-a721550149a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" Oct 31 14:00:00.538874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.537144 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m2p7\" (UniqueName: \"kubernetes.io/projected/27c88326-5db1-4240-bbb5-75b2ed7a72a9-kube-api-access-6m2p7\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8\" (UID: \"27c88326-5db1-4240-bbb5-75b2ed7a72a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" Oct 31 14:00:00.538874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.537197 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x9j7\" (UniqueName: \"kubernetes.io/projected/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-kube-api-access-2x9j7\") pod \"maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2\" (UID: \"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" Oct 31 14:00:00.538874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.537341 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c88326-5db1-4240-bbb5-75b2ed7a72a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8\" (UID: \"27c88326-5db1-4240-bbb5-75b2ed7a72a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" Oct 31 14:00:00.538874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.537465 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2\" (UID: \"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" Oct 31 14:00:00.538874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.537521 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d6b4fa-53be-495d-ba97-f8edf753caa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890920-f5wbl\" (UID: \"d2d6b4fa-53be-495d-ba97-f8edf753caa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" Oct 31 14:00:00.538874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.537567 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w5mq\" (UniqueName: \"kubernetes.io/projected/641d472a-fcc8-4c9c-bac0-a721550149a0-kube-api-access-4w5mq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54\" (UID: \"641d472a-fcc8-4c9c-bac0-a721550149a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" Oct 31 14:00:00.538874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.538585 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2\" (UID: \"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" Oct 31 14:00:00.538874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.538789 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d6b4fa-53be-495d-ba97-f8edf753caa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890920-f5wbl\" (UID: \"d2d6b4fa-53be-495d-ba97-f8edf753caa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" Oct 31 14:00:00.539580 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.539138 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641d472a-fcc8-4c9c-bac0-a721550149a0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54\" (UID: \"641d472a-fcc8-4c9c-bac0-a721550149a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" Oct 31 14:00:00.541768 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.541743 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c88326-5db1-4240-bbb5-75b2ed7a72a9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8\" (UID: \"27c88326-5db1-4240-bbb5-75b2ed7a72a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" Oct 31 14:00:00.557997 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.556951 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqzcf\" (UniqueName: \"kubernetes.io/projected/d2d6b4fa-53be-495d-ba97-f8edf753caa4-kube-api-access-nqzcf\") pod \"maas-api-key-cleanup-29890920-f5wbl\" (UID: \"d2d6b4fa-53be-495d-ba97-f8edf753caa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" Oct 31 14:00:00.557997 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.557930 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x9j7\" (UniqueName: \"kubernetes.io/projected/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-kube-api-access-2x9j7\") pod \"maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2\" (UID: \"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" Oct 31 14:00:00.558876 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.558847 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w5mq\" (UniqueName: \"kubernetes.io/projected/641d472a-fcc8-4c9c-bac0-a721550149a0-kube-api-access-4w5mq\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54\" (UID: \"641d472a-fcc8-4c9c-bac0-a721550149a0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" Oct 31 14:00:00.558989 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.558911 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m2p7\" (UniqueName: \"kubernetes.io/projected/27c88326-5db1-4240-bbb5-75b2ed7a72a9-kube-api-access-6m2p7\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8\" (UID: \"27c88326-5db1-4240-bbb5-75b2ed7a72a9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" Oct 31 14:00:00.561654 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.561626 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" Oct 31 14:00:00.582284 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.581934 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" Oct 31 14:00:00.586358 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.586028 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" Oct 31 14:00:00.600480 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.599666 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" Oct 31 14:00:00.612260 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.612060 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" Oct 31 14:00:00.625418 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.625391 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" Oct 31 14:00:00.641247 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.641035 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" Oct 31 14:00:00.790204 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:00.790173 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv"] Oct 31 14:00:00.795479 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:00:00.795433 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5bd142e_0a36_4389_a6f7_da08f0e4c77a.slice/crio-ac41193139ca6a146212d3e5dd83bb9deb024fc9a085658c19a69e604ceb95b8 WatchSource:0}: Error finding container ac41193139ca6a146212d3e5dd83bb9deb024fc9a085658c19a69e604ceb95b8: Status 404 returned error can't find the container with id ac41193139ca6a146212d3e5dd83bb9deb024fc9a085658c19a69e604ceb95b8 Oct 31 14:00:01.036486 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.036367 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8"] Oct 31 14:00:01.053439 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.053411 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv"] Oct 31 14:00:01.055826 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:00:01.055790 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f8ff8c_ffda_4af4_bc4d_7795c623ddef.slice/crio-95b7a5b2adb201c1dcfb0d0aef20f8e90d417f62db3d1bde5b3f82ea64c772f6 WatchSource:0}: Error finding container 95b7a5b2adb201c1dcfb0d0aef20f8e90d417f62db3d1bde5b3f82ea64c772f6: Status 404 returned error can't find the container with id 95b7a5b2adb201c1dcfb0d0aef20f8e90d417f62db3d1bde5b3f82ea64c772f6 Oct 31 14:00:01.079691 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.071641 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl"] Oct 31 14:00:01.094533 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.094493 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5"] Oct 31 14:00:01.156885 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.156864 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr"] Oct 31 14:00:01.161100 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.161076 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4"] Oct 31 14:00:01.197164 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.197141 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8"] Oct 31 14:00:01.201114 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.201093 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54"] Oct 31 14:00:01.238830 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.238807 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2"] Oct 31 14:00:01.258115 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:00:01.258050 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ab1aa4_e729_443c_a322_1a4ce645be61.slice/crio-b8e0d7431a37e21d5b210a514961e1bf97ed0510fe1e9f1f2fb11161782c90a0 WatchSource:0}: Error finding container b8e0d7431a37e21d5b210a514961e1bf97ed0510fe1e9f1f2fb11161782c90a0: Status 404 returned error can't find the container with id b8e0d7431a37e21d5b210a514961e1bf97ed0510fe1e9f1f2fb11161782c90a0 Oct 31 14:00:01.628232 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.628199 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" event={"ID":"641d472a-fcc8-4c9c-bac0-a721550149a0","Type":"ContainerStarted","Data":"4a9ddeb5f41179b8231245ba44a5b803ae4c3ea1af8216bf7111fe5a9235a21d"} Oct 31 14:00:01.629585 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.629561 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" event={"ID":"4a6c45b7-8d74-481a-b429-cc7c65420bdd","Type":"ContainerStarted","Data":"9b9b51041e3577a6a48403c97fc918e26bf08567bcfc2d29182b2178374830da"} Oct 31 14:00:01.630836 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.630811 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" event={"ID":"27c88326-5db1-4240-bbb5-75b2ed7a72a9","Type":"ContainerStarted","Data":"a866f670eb0b893f6f92c7adbcaf164dfa7a7f024a13f41704c0ff4e1acad64b"} Oct 31 14:00:01.632077 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.632054 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" event={"ID":"65744c55-b224-4b04-8705-b26423016c51","Type":"ContainerStarted","Data":"ed4f3d423fc1a817d8e2ab18b6448820649683e72a371ef3ae18dc456d55dfe7"} Oct 31 14:00:01.633489 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.633465 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" event={"ID":"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef","Type":"ContainerStarted","Data":"a4adad30283d257bc181e3fde37288a2b432a3658720b28684c1d04dc9327301"} Oct 31 14:00:01.633572 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.633511 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" event={"ID":"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef","Type":"ContainerStarted","Data":"95b7a5b2adb201c1dcfb0d0aef20f8e90d417f62db3d1bde5b3f82ea64c772f6"} Oct 31 14:00:01.635179 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.635143 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" event={"ID":"d2d6b4fa-53be-495d-ba97-f8edf753caa4","Type":"ContainerStarted","Data":"4a9140ef2388869c9eaa9cf5311460092641dfa880488c7f6248daa36278f4bf"} Oct 31 14:00:01.635268 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.635190 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" event={"ID":"d2d6b4fa-53be-495d-ba97-f8edf753caa4","Type":"ContainerStarted","Data":"3b9e0f56d4a7e938d4f178c1c0c0998daad8bcdf592f71784ed3ac7e37f6f20c"} Oct 31 14:00:01.636657 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.636627 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" event={"ID":"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf","Type":"ContainerStarted","Data":"6b17432cc2e186775464521f03f1bab10527c0f9ed234550c0a44b486c7f84fa"} Oct 31 14:00:01.638371 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.638342 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" event={"ID":"d5bd142e-0a36-4389-a6f7-da08f0e4c77a","Type":"ContainerStarted","Data":"80424e54821d3d1bf12811a150e81a4df667b77aa1752f152291812946265ec2"} Oct 31 14:00:01.638507 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.638392 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" event={"ID":"d5bd142e-0a36-4389-a6f7-da08f0e4c77a","Type":"ContainerStarted","Data":"ac41193139ca6a146212d3e5dd83bb9deb024fc9a085658c19a69e604ceb95b8"} Oct 31 14:00:01.640176 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.640153 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" event={"ID":"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed","Type":"ContainerStarted","Data":"ff806ad48357ee97d06a070b01613156c3dc4880342a1822e1834c31ddfbb60c"} Oct 31 14:00:01.641537 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.641515 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" event={"ID":"d5ab1aa4-e729-443c-a322-1a4ce645be61","Type":"ContainerStarted","Data":"b8e0d7431a37e21d5b210a514961e1bf97ed0510fe1e9f1f2fb11161782c90a0"} Oct 31 14:00:01.663015 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.662964 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" podStartSLOduration=1.6629503140000002 podStartE2EDuration="1.662950314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:01.661715736 +0000 UTC m=+3369.807459564" watchObservedRunningTime="2026-10-31 14:00:01.662950314 +0000 UTC m=+3369.808694140" Oct 31 14:00:01.685936 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.685889 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" podStartSLOduration=1.685875499 podStartE2EDuration="1.685875499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:01.681539523 +0000 UTC m=+3369.827283353" watchObservedRunningTime="2026-10-31 14:00:01.685875499 +0000 UTC m=+3369.831619327" Oct 31 14:00:01.704207 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:01.704151 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" podStartSLOduration=1.7041164119999999 podStartE2EDuration="1.704116412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:01.700659417 +0000 UTC m=+3369.846403244" watchObservedRunningTime="2026-10-31 14:00:01.704116412 +0000 UTC m=+3369.849860243" Oct 31 14:00:01.944572 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:01.944509 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:00:02.653727 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.652930 5532 generic.go:356] "Generic (PLEG): container finished" podID="9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed" containerID="6ef1c97b0f9df26b2bd3d07030eb6de0ffa77aa99bce64d54c250eec9f77da93" exitCode=60 Oct 31 14:00:02.653727 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.653030 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" event={"ID":"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed","Type":"ContainerDied","Data":"6ef1c97b0f9df26b2bd3d07030eb6de0ffa77aa99bce64d54c250eec9f77da93"} Oct 31 14:00:02.653727 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.653448 5532 scope.go:122] "RemoveContainer" containerID="6ef1c97b0f9df26b2bd3d07030eb6de0ffa77aa99bce64d54c250eec9f77da93" Oct 31 14:00:02.656014 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.655976 5532 generic.go:356] "Generic (PLEG): container finished" podID="d5ab1aa4-e729-443c-a322-1a4ce645be61" containerID="286f191f7d1c9af8a6e18c30da41c6fbe0ecdda5d8bbf632698dc2fefa59da67" exitCode=60 Oct 31 14:00:02.656156 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.656060 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" event={"ID":"d5ab1aa4-e729-443c-a322-1a4ce645be61","Type":"ContainerDied","Data":"286f191f7d1c9af8a6e18c30da41c6fbe0ecdda5d8bbf632698dc2fefa59da67"} Oct 31 14:00:02.656351 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.656335 5532 scope.go:122] "RemoveContainer" containerID="286f191f7d1c9af8a6e18c30da41c6fbe0ecdda5d8bbf632698dc2fefa59da67" Oct 31 14:00:02.658669 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.658645 5532 generic.go:356] "Generic (PLEG): container finished" podID="641d472a-fcc8-4c9c-bac0-a721550149a0" containerID="8b3bf7adbc37b77038b01bff897d63374dd96f09be7ffba25e6e642cf3c1a649" exitCode=60 Oct 31 14:00:02.658980 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.658725 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" event={"ID":"641d472a-fcc8-4c9c-bac0-a721550149a0","Type":"ContainerDied","Data":"8b3bf7adbc37b77038b01bff897d63374dd96f09be7ffba25e6e642cf3c1a649"} Oct 31 14:00:02.659074 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.659019 5532 scope.go:122] "RemoveContainer" containerID="8b3bf7adbc37b77038b01bff897d63374dd96f09be7ffba25e6e642cf3c1a649" Oct 31 14:00:02.662420 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.662394 5532 generic.go:356] "Generic (PLEG): container finished" podID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" containerID="d7b2d54107f1eb63d20cdf8b38218687ebd71ae69918ef4590bc6dc9c6886086" exitCode=60 Oct 31 14:00:02.662533 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.662502 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" event={"ID":"4a6c45b7-8d74-481a-b429-cc7c65420bdd","Type":"ContainerDied","Data":"d7b2d54107f1eb63d20cdf8b38218687ebd71ae69918ef4590bc6dc9c6886086"} Oct 31 14:00:02.662756 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.662734 5532 scope.go:122] "RemoveContainer" containerID="d7b2d54107f1eb63d20cdf8b38218687ebd71ae69918ef4590bc6dc9c6886086" Oct 31 14:00:02.664502 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.664474 5532 generic.go:356] "Generic (PLEG): container finished" podID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" containerID="bfc3818885f6f53061a7d9a7f5a08aa578136ed403ca8b92c6fb356b209c162d" exitCode=60 Oct 31 14:00:02.664648 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.664565 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" event={"ID":"27c88326-5db1-4240-bbb5-75b2ed7a72a9","Type":"ContainerDied","Data":"bfc3818885f6f53061a7d9a7f5a08aa578136ed403ca8b92c6fb356b209c162d"} Oct 31 14:00:02.664864 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.664847 5532 scope.go:122] "RemoveContainer" containerID="bfc3818885f6f53061a7d9a7f5a08aa578136ed403ca8b92c6fb356b209c162d" Oct 31 14:00:02.666513 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.666487 5532 generic.go:356] "Generic (PLEG): container finished" podID="65744c55-b224-4b04-8705-b26423016c51" containerID="e8a6a019d67b806d45e3e9bdd11e8f6b5e082bbd620d1c7f952c66dd83bf3d9e" exitCode=60 Oct 31 14:00:02.666651 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.666555 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" event={"ID":"65744c55-b224-4b04-8705-b26423016c51","Type":"ContainerDied","Data":"e8a6a019d67b806d45e3e9bdd11e8f6b5e082bbd620d1c7f952c66dd83bf3d9e"} Oct 31 14:00:02.666904 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.666885 5532 scope.go:122] "RemoveContainer" containerID="e8a6a019d67b806d45e3e9bdd11e8f6b5e082bbd620d1c7f952c66dd83bf3d9e" Oct 31 14:00:02.668490 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.668468 5532 generic.go:356] "Generic (PLEG): container finished" podID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" containerID="a4adad30283d257bc181e3fde37288a2b432a3658720b28684c1d04dc9327301" exitCode=60 Oct 31 14:00:02.668581 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.668570 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" event={"ID":"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef","Type":"ContainerDied","Data":"a4adad30283d257bc181e3fde37288a2b432a3658720b28684c1d04dc9327301"} Oct 31 14:00:02.668951 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.668927 5532 scope.go:122] "RemoveContainer" containerID="a4adad30283d257bc181e3fde37288a2b432a3658720b28684c1d04dc9327301" Oct 31 14:00:02.670989 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.670968 5532 generic.go:356] "Generic (PLEG): container finished" podID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" containerID="4a9140ef2388869c9eaa9cf5311460092641dfa880488c7f6248daa36278f4bf" exitCode=60 Oct 31 14:00:02.671069 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.671033 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" event={"ID":"d2d6b4fa-53be-495d-ba97-f8edf753caa4","Type":"ContainerDied","Data":"4a9140ef2388869c9eaa9cf5311460092641dfa880488c7f6248daa36278f4bf"} Oct 31 14:00:02.671257 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.671243 5532 scope.go:122] "RemoveContainer" containerID="4a9140ef2388869c9eaa9cf5311460092641dfa880488c7f6248daa36278f4bf" Oct 31 14:00:02.676320 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.676183 5532 generic.go:356] "Generic (PLEG): container finished" podID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" containerID="ede8de1523b9cb0902f39f143dd2fb5844a73c1ca28bf46a6e0bff31a73a6a67" exitCode=60 Oct 31 14:00:02.676320 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.676262 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" event={"ID":"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf","Type":"ContainerDied","Data":"ede8de1523b9cb0902f39f143dd2fb5844a73c1ca28bf46a6e0bff31a73a6a67"} Oct 31 14:00:02.676819 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.676528 5532 scope.go:122] "RemoveContainer" containerID="ede8de1523b9cb0902f39f143dd2fb5844a73c1ca28bf46a6e0bff31a73a6a67" Oct 31 14:00:02.678684 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.678438 5532 generic.go:356] "Generic (PLEG): container finished" podID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" containerID="80424e54821d3d1bf12811a150e81a4df667b77aa1752f152291812946265ec2" exitCode=60 Oct 31 14:00:02.678684 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.678483 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" event={"ID":"d5bd142e-0a36-4389-a6f7-da08f0e4c77a","Type":"ContainerDied","Data":"80424e54821d3d1bf12811a150e81a4df667b77aa1752f152291812946265ec2"} Oct 31 14:00:02.678929 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:02.678917 5532 scope.go:122] "RemoveContainer" containerID="80424e54821d3d1bf12811a150e81a4df667b77aa1752f152291812946265ec2" Oct 31 14:00:03.696124 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.696089 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" event={"ID":"65744c55-b224-4b04-8705-b26423016c51","Type":"ContainerStarted","Data":"46b57a2a4cc81532fa49c479aefb1686dc13fe44b4f77a9b35c5f80e59c249f9"} Oct 31 14:00:03.698355 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.698325 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" event={"ID":"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef","Type":"ContainerStarted","Data":"01e71f9d6eafb536489845cdf390e88eff83f7b5baf7e202e4cf6e5dc9f7f0cb"} Oct 31 14:00:03.700536 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.700508 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" event={"ID":"d2d6b4fa-53be-495d-ba97-f8edf753caa4","Type":"ContainerStarted","Data":"6f5d7a32001cebfb5bd1a7d37c3bda8d640aea25c6f4fc2cd4bdf25ac9f5bbd9"} Oct 31 14:00:03.702755 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.702729 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" event={"ID":"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf","Type":"ContainerStarted","Data":"242fbcb369f73cbde97993a5b5e301fd9191d55b6b6f8eed1e43b83b34de3397"} Oct 31 14:00:03.704886 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.704863 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" event={"ID":"d5bd142e-0a36-4389-a6f7-da08f0e4c77a","Type":"ContainerStarted","Data":"de47cbc9f1413a0ce05b3c35262fbe97866ae0b70c91fe1c70e211bdbfb2aab1"} Oct 31 14:00:03.706904 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.706883 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" event={"ID":"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed","Type":"ContainerStarted","Data":"2ab743c2bc4a1724bbd413c2ca9c75238b2e858d3c187b4ddda22ae92597c5ff"} Oct 31 14:00:03.708860 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.708840 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" event={"ID":"d5ab1aa4-e729-443c-a322-1a4ce645be61","Type":"ContainerStarted","Data":"39a0a46a17adcc30b535d0b56a834275f02b36b25553dbf7a0fee76385015539"} Oct 31 14:00:03.710639 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.710612 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" event={"ID":"641d472a-fcc8-4c9c-bac0-a721550149a0","Type":"ContainerStarted","Data":"3800a9b301317ef03ab85d03481f6bf8480d60566251d817389aa2ffd1eff9e8"} Oct 31 14:00:03.712445 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.712423 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" event={"ID":"4a6c45b7-8d74-481a-b429-cc7c65420bdd","Type":"ContainerStarted","Data":"bb3471275137564ffb0c949fe3d67c9e9e08dacb93c1f470c3def864f000d867"} Oct 31 14:00:03.714323 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.714292 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" event={"ID":"27c88326-5db1-4240-bbb5-75b2ed7a72a9","Type":"ContainerStarted","Data":"e472bb319ee5df00cc6792883d70ac9192a2cbc9df5fa7a6d56a0cf859a3cf78"} Oct 31 14:00:03.719460 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.719413 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" podStartSLOduration=3.719400888 podStartE2EDuration="3.719400888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:03.714623794 +0000 UTC m=+3371.860367622" watchObservedRunningTime="2026-10-31 14:00:03.719400888 +0000 UTC m=+3371.865144715" Oct 31 14:00:03.780815 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.780740 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" podStartSLOduration=3.780718345 podStartE2EDuration="3.780718345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:03.76423215 +0000 UTC m=+3371.909975978" watchObservedRunningTime="2026-10-31 14:00:03.780718345 +0000 UTC m=+3371.926462175" Oct 31 14:00:03.819741 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.819624 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" podStartSLOduration=3.819605688 podStartE2EDuration="3.819605688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:03.794957545 +0000 UTC m=+3371.940701375" watchObservedRunningTime="2026-10-31 14:00:03.819605688 +0000 UTC m=+3371.965349510" Oct 31 14:00:03.846831 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.846778 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" podStartSLOduration=3.846762904 podStartE2EDuration="3.846762904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:03.845160754 +0000 UTC m=+3371.990904583" watchObservedRunningTime="2026-10-31 14:00:03.846762904 +0000 UTC m=+3371.992506777" Oct 31 14:00:03.874248 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.874194 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" podStartSLOduration=3.874180941 podStartE2EDuration="3.874180941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:03.869735593 +0000 UTC m=+3372.015479416" watchObservedRunningTime="2026-10-31 14:00:03.874180941 +0000 UTC m=+3372.019924768" Oct 31 14:00:03.913954 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.913884 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" podStartSLOduration=3.913869246 podStartE2EDuration="3.913869246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:03.910146104 +0000 UTC m=+3372.055889933" watchObservedRunningTime="2026-10-31 14:00:03.913869246 +0000 UTC m=+3372.059613102" Oct 31 14:00:03.933258 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:03.933190 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" podStartSLOduration=3.933172172 podStartE2EDuration="3.933172172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:03.93217828 +0000 UTC m=+3372.077922111" watchObservedRunningTime="2026-10-31 14:00:03.933172172 +0000 UTC m=+3372.078916002" Oct 31 14:00:04.725970 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.725931 5532 generic.go:356] "Generic (PLEG): container finished" podID="65744c55-b224-4b04-8705-b26423016c51" containerID="46b57a2a4cc81532fa49c479aefb1686dc13fe44b4f77a9b35c5f80e59c249f9" exitCode=60 Oct 31 14:00:04.726457 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.726016 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" event={"ID":"65744c55-b224-4b04-8705-b26423016c51","Type":"ContainerDied","Data":"46b57a2a4cc81532fa49c479aefb1686dc13fe44b4f77a9b35c5f80e59c249f9"} Oct 31 14:00:04.726457 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.726086 5532 scope.go:122] "RemoveContainer" containerID="e8a6a019d67b806d45e3e9bdd11e8f6b5e082bbd620d1c7f952c66dd83bf3d9e" Oct 31 14:00:04.726457 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.726359 5532 scope.go:122] "RemoveContainer" containerID="46b57a2a4cc81532fa49c479aefb1686dc13fe44b4f77a9b35c5f80e59c249f9" Oct 31 14:00:04.726814 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:04.726782 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5_odh-ai-gateway-infra(65744c55-b224-4b04-8705-b26423016c51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" podUID="65744c55-b224-4b04-8705-b26423016c51" Oct 31 14:00:04.728456 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.728433 5532 generic.go:356] "Generic (PLEG): container finished" podID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" containerID="01e71f9d6eafb536489845cdf390e88eff83f7b5baf7e202e4cf6e5dc9f7f0cb" exitCode=60 Oct 31 14:00:04.728556 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.728503 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" event={"ID":"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef","Type":"ContainerDied","Data":"01e71f9d6eafb536489845cdf390e88eff83f7b5baf7e202e4cf6e5dc9f7f0cb"} Oct 31 14:00:04.728846 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.728827 5532 scope.go:122] "RemoveContainer" containerID="01e71f9d6eafb536489845cdf390e88eff83f7b5baf7e202e4cf6e5dc9f7f0cb" Oct 31 14:00:04.729126 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:04.729102 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv_odh-ai-gateway-infra(f5f8ff8c-ffda-4af4-bc4d-7795c623ddef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" podUID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" Oct 31 14:00:04.730546 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.730524 5532 generic.go:356] "Generic (PLEG): container finished" podID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" containerID="6f5d7a32001cebfb5bd1a7d37c3bda8d640aea25c6f4fc2cd4bdf25ac9f5bbd9" exitCode=60 Oct 31 14:00:04.730688 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.730631 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" event={"ID":"d2d6b4fa-53be-495d-ba97-f8edf753caa4","Type":"ContainerDied","Data":"6f5d7a32001cebfb5bd1a7d37c3bda8d640aea25c6f4fc2cd4bdf25ac9f5bbd9"} Oct 31 14:00:04.730879 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.730862 5532 scope.go:122] "RemoveContainer" containerID="6f5d7a32001cebfb5bd1a7d37c3bda8d640aea25c6f4fc2cd4bdf25ac9f5bbd9" Oct 31 14:00:04.731177 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:04.731159 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890920-f5wbl_odh-ai-gateway-infra(d2d6b4fa-53be-495d-ba97-f8edf753caa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" podUID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" Oct 31 14:00:04.732643 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.732623 5532 generic.go:356] "Generic (PLEG): container finished" podID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" containerID="242fbcb369f73cbde97993a5b5e301fd9191d55b6b6f8eed1e43b83b34de3397" exitCode=60 Oct 31 14:00:04.732733 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.732702 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" event={"ID":"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf","Type":"ContainerDied","Data":"242fbcb369f73cbde97993a5b5e301fd9191d55b6b6f8eed1e43b83b34de3397"} Oct 31 14:00:04.732949 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.732931 5532 scope.go:122] "RemoveContainer" containerID="242fbcb369f73cbde97993a5b5e301fd9191d55b6b6f8eed1e43b83b34de3397" Oct 31 14:00:04.733272 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:04.733246 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr_odh-ai-gateway-infra(a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" podUID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" Oct 31 14:00:04.735507 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.734873 5532 generic.go:356] "Generic (PLEG): container finished" podID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" containerID="de47cbc9f1413a0ce05b3c35262fbe97866ae0b70c91fe1c70e211bdbfb2aab1" exitCode=60 Oct 31 14:00:04.735507 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.735033 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" event={"ID":"d5bd142e-0a36-4389-a6f7-da08f0e4c77a","Type":"ContainerDied","Data":"de47cbc9f1413a0ce05b3c35262fbe97866ae0b70c91fe1c70e211bdbfb2aab1"} Oct 31 14:00:04.735507 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.735154 5532 scope.go:122] "RemoveContainer" containerID="de47cbc9f1413a0ce05b3c35262fbe97866ae0b70c91fe1c70e211bdbfb2aab1" Oct 31 14:00:04.735507 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:04.735380 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv_odh-ai-gateway-infra(d5bd142e-0a36-4389-a6f7-da08f0e4c77a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" podUID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" Oct 31 14:00:04.737185 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.737166 5532 generic.go:356] "Generic (PLEG): container finished" podID="9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed" containerID="2ab743c2bc4a1724bbd413c2ca9c75238b2e858d3c187b4ddda22ae92597c5ff" exitCode=60 Oct 31 14:00:04.737281 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.737235 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" event={"ID":"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed","Type":"ContainerDied","Data":"2ab743c2bc4a1724bbd413c2ca9c75238b2e858d3c187b4ddda22ae92597c5ff"} Oct 31 14:00:04.737487 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.737472 5532 scope.go:122] "RemoveContainer" containerID="2ab743c2bc4a1724bbd413c2ca9c75238b2e858d3c187b4ddda22ae92597c5ff" Oct 31 14:00:04.737776 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:04.737753 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2_odh-ai-gateway-infra(9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" podUID="9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed" Oct 31 14:00:04.739575 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.739546 5532 generic.go:356] "Generic (PLEG): container finished" podID="d5ab1aa4-e729-443c-a322-1a4ce645be61" containerID="39a0a46a17adcc30b535d0b56a834275f02b36b25553dbf7a0fee76385015539" exitCode=60 Oct 31 14:00:04.739680 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.739650 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" event={"ID":"d5ab1aa4-e729-443c-a322-1a4ce645be61","Type":"ContainerDied","Data":"39a0a46a17adcc30b535d0b56a834275f02b36b25553dbf7a0fee76385015539"} Oct 31 14:00:04.739905 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.739888 5532 scope.go:122] "RemoveContainer" containerID="39a0a46a17adcc30b535d0b56a834275f02b36b25553dbf7a0fee76385015539" Oct 31 14:00:04.740168 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:04.740142 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4_odh-ai-gateway-infra(d5ab1aa4-e729-443c-a322-1a4ce645be61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" podUID="d5ab1aa4-e729-443c-a322-1a4ce645be61" Oct 31 14:00:04.741862 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.741837 5532 generic.go:356] "Generic (PLEG): container finished" podID="641d472a-fcc8-4c9c-bac0-a721550149a0" containerID="3800a9b301317ef03ab85d03481f6bf8480d60566251d817389aa2ffd1eff9e8" exitCode=60 Oct 31 14:00:04.741948 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.741916 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" event={"ID":"641d472a-fcc8-4c9c-bac0-a721550149a0","Type":"ContainerDied","Data":"3800a9b301317ef03ab85d03481f6bf8480d60566251d817389aa2ffd1eff9e8"} Oct 31 14:00:04.742139 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.742121 5532 scope.go:122] "RemoveContainer" containerID="3800a9b301317ef03ab85d03481f6bf8480d60566251d817389aa2ffd1eff9e8" Oct 31 14:00:04.742404 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:04.742380 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54_odh-ai-gateway-infra(641d472a-fcc8-4c9c-bac0-a721550149a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" podUID="641d472a-fcc8-4c9c-bac0-a721550149a0" Oct 31 14:00:04.743648 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.743584 5532 scope.go:122] "RemoveContainer" containerID="a4adad30283d257bc181e3fde37288a2b432a3658720b28684c1d04dc9327301" Oct 31 14:00:04.744114 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.744089 5532 generic.go:356] "Generic (PLEG): container finished" podID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" containerID="bb3471275137564ffb0c949fe3d67c9e9e08dacb93c1f470c3def864f000d867" exitCode=60 Oct 31 14:00:04.744208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.744164 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" event={"ID":"4a6c45b7-8d74-481a-b429-cc7c65420bdd","Type":"ContainerDied","Data":"bb3471275137564ffb0c949fe3d67c9e9e08dacb93c1f470c3def864f000d867"} Oct 31 14:00:04.745634 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.744406 5532 scope.go:122] "RemoveContainer" containerID="bb3471275137564ffb0c949fe3d67c9e9e08dacb93c1f470c3def864f000d867" Oct 31 14:00:04.745634 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:04.744804 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8_odh-ai-gateway-infra(4a6c45b7-8d74-481a-b429-cc7c65420bdd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" podUID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" Oct 31 14:00:04.747510 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.747491 5532 generic.go:356] "Generic (PLEG): container finished" podID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" containerID="e472bb319ee5df00cc6792883d70ac9192a2cbc9df5fa7a6d56a0cf859a3cf78" exitCode=60 Oct 31 14:00:04.747582 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.747537 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" event={"ID":"27c88326-5db1-4240-bbb5-75b2ed7a72a9","Type":"ContainerDied","Data":"e472bb319ee5df00cc6792883d70ac9192a2cbc9df5fa7a6d56a0cf859a3cf78"} Oct 31 14:00:04.747807 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.747793 5532 scope.go:122] "RemoveContainer" containerID="e472bb319ee5df00cc6792883d70ac9192a2cbc9df5fa7a6d56a0cf859a3cf78" Oct 31 14:00:04.748012 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:04.747987 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8_odh-ai-gateway-infra(27c88326-5db1-4240-bbb5-75b2ed7a72a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" podUID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" Oct 31 14:00:04.759309 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.759287 5532 scope.go:122] "RemoveContainer" containerID="4a9140ef2388869c9eaa9cf5311460092641dfa880488c7f6248daa36278f4bf" Oct 31 14:00:04.774516 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.774496 5532 scope.go:122] "RemoveContainer" containerID="ede8de1523b9cb0902f39f143dd2fb5844a73c1ca28bf46a6e0bff31a73a6a67" Oct 31 14:00:04.791890 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.791868 5532 scope.go:122] "RemoveContainer" containerID="80424e54821d3d1bf12811a150e81a4df667b77aa1752f152291812946265ec2" Oct 31 14:00:04.807075 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.807052 5532 scope.go:122] "RemoveContainer" containerID="6ef1c97b0f9df26b2bd3d07030eb6de0ffa77aa99bce64d54c250eec9f77da93" Oct 31 14:00:04.830765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.830744 5532 scope.go:122] "RemoveContainer" containerID="286f191f7d1c9af8a6e18c30da41c6fbe0ecdda5d8bbf632698dc2fefa59da67" Oct 31 14:00:04.852635 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.852572 5532 scope.go:122] "RemoveContainer" containerID="8b3bf7adbc37b77038b01bff897d63374dd96f09be7ffba25e6e642cf3c1a649" Oct 31 14:00:04.871868 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.871844 5532 scope.go:122] "RemoveContainer" containerID="d7b2d54107f1eb63d20cdf8b38218687ebd71ae69918ef4590bc6dc9c6886086" Oct 31 14:00:04.886785 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:04.886764 5532 scope.go:122] "RemoveContainer" containerID="bfc3818885f6f53061a7d9a7f5a08aa578136ed403ca8b92c6fb356b209c162d" Oct 31 14:00:05.761802 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:05.761772 5532 scope.go:122] "RemoveContainer" containerID="2ab743c2bc4a1724bbd413c2ca9c75238b2e858d3c187b4ddda22ae92597c5ff" Oct 31 14:00:05.762325 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:05.762175 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2_odh-ai-gateway-infra(9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" podUID="9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed" Oct 31 14:00:05.763641 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:05.763584 5532 scope.go:122] "RemoveContainer" containerID="39a0a46a17adcc30b535d0b56a834275f02b36b25553dbf7a0fee76385015539" Oct 31 14:00:05.763998 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:05.763976 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4_odh-ai-gateway-infra(d5ab1aa4-e729-443c-a322-1a4ce645be61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" podUID="d5ab1aa4-e729-443c-a322-1a4ce645be61" Oct 31 14:00:05.765476 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:05.765455 5532 scope.go:122] "RemoveContainer" containerID="3800a9b301317ef03ab85d03481f6bf8480d60566251d817389aa2ffd1eff9e8" Oct 31 14:00:05.765795 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:05.765772 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54_odh-ai-gateway-infra(641d472a-fcc8-4c9c-bac0-a721550149a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" podUID="641d472a-fcc8-4c9c-bac0-a721550149a0" Oct 31 14:00:05.767351 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:05.767322 5532 scope.go:122] "RemoveContainer" containerID="bb3471275137564ffb0c949fe3d67c9e9e08dacb93c1f470c3def864f000d867" Oct 31 14:00:05.767706 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:05.767685 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8_odh-ai-gateway-infra(4a6c45b7-8d74-481a-b429-cc7c65420bdd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" podUID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" Oct 31 14:00:05.769079 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:05.769061 5532 scope.go:122] "RemoveContainer" containerID="e472bb319ee5df00cc6792883d70ac9192a2cbc9df5fa7a6d56a0cf859a3cf78" Oct 31 14:00:05.769365 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:05.769338 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8_odh-ai-gateway-infra(27c88326-5db1-4240-bbb5-75b2ed7a72a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" podUID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" Oct 31 14:00:05.770755 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:05.770737 5532 scope.go:122] "RemoveContainer" containerID="46b57a2a4cc81532fa49c479aefb1686dc13fe44b4f77a9b35c5f80e59c249f9" Oct 31 14:00:05.771021 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:05.771000 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5_odh-ai-gateway-infra(65744c55-b224-4b04-8705-b26423016c51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" podUID="65744c55-b224-4b04-8705-b26423016c51" Oct 31 14:00:05.772505 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:05.772484 5532 scope.go:122] "RemoveContainer" containerID="01e71f9d6eafb536489845cdf390e88eff83f7b5baf7e202e4cf6e5dc9f7f0cb" Oct 31 14:00:05.772787 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:05.772763 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv_odh-ai-gateway-infra(f5f8ff8c-ffda-4af4-bc4d-7795c623ddef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" podUID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" Oct 31 14:00:05.774216 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:05.774199 5532 scope.go:122] "RemoveContainer" containerID="6f5d7a32001cebfb5bd1a7d37c3bda8d640aea25c6f4fc2cd4bdf25ac9f5bbd9" Oct 31 14:00:05.774477 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:05.774459 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890920-f5wbl_odh-ai-gateway-infra(d2d6b4fa-53be-495d-ba97-f8edf753caa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" podUID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" Oct 31 14:00:05.776006 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:05.775990 5532 scope.go:122] "RemoveContainer" containerID="242fbcb369f73cbde97993a5b5e301fd9191d55b6b6f8eed1e43b83b34de3397" Oct 31 14:00:05.776268 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:05.776241 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr_odh-ai-gateway-infra(a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" podUID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" Oct 31 14:00:05.777643 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:05.777625 5532 scope.go:122] "RemoveContainer" containerID="de47cbc9f1413a0ce05b3c35262fbe97866ae0b70c91fe1c70e211bdbfb2aab1" Oct 31 14:00:05.777857 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:05.777831 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv_odh-ai-gateway-infra(d5bd142e-0a36-4389-a6f7-da08f0e4c77a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" podUID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" Oct 31 14:00:07.237393 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:07.237359 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc"] Oct 31 14:00:07.243651 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:07.243628 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" Oct 31 14:00:07.250454 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:07.250426 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f9ca72\"" Oct 31 14:00:07.250638 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:07.250503 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f9ca72-dockercfg-j6qhd\"" Oct 31 14:00:07.254641 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:07.253606 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc"] Oct 31 14:00:07.379614 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:07.371336 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc\" (UID: \"a5959be5-1f14-4ea5-a4b0-8d6b141052f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" Oct 31 14:00:07.379614 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:07.371419 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqskh\" (UniqueName: \"kubernetes.io/projected/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-kube-api-access-lqskh\") pod \"payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc\" (UID: \"a5959be5-1f14-4ea5-a4b0-8d6b141052f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" Oct 31 14:00:07.492713 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:07.485573 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc\" (UID: \"a5959be5-1f14-4ea5-a4b0-8d6b141052f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" Oct 31 14:00:07.492713 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:07.485678 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqskh\" (UniqueName: \"kubernetes.io/projected/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-kube-api-access-lqskh\") pod \"payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc\" (UID: \"a5959be5-1f14-4ea5-a4b0-8d6b141052f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" Oct 31 14:00:07.492713 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:07.487410 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc\" (UID: \"a5959be5-1f14-4ea5-a4b0-8d6b141052f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" Oct 31 14:00:07.504499 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:07.504460 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqskh\" (UniqueName: \"kubernetes.io/projected/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-kube-api-access-lqskh\") pod \"payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc\" (UID: \"a5959be5-1f14-4ea5-a4b0-8d6b141052f2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" Oct 31 14:00:07.563033 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:07.562994 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" Oct 31 14:00:07.850158 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:00:07.850126 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5959be5_1f14_4ea5_a4b0_8d6b141052f2.slice/crio-6ac2669b761080778858060f8a1c4c77e321eba21fb121c666d45efc3c2e9273 WatchSource:0}: Error finding container 6ac2669b761080778858060f8a1c4c77e321eba21fb121c666d45efc3c2e9273: Status 404 returned error can't find the container with id 6ac2669b761080778858060f8a1c4c77e321eba21fb121c666d45efc3c2e9273 Oct 31 14:00:07.850490 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:07.850468 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc"] Oct 31 14:00:08.806686 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:08.806142 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" event={"ID":"a5959be5-1f14-4ea5-a4b0-8d6b141052f2","Type":"ContainerStarted","Data":"d78689cfce4c0a281d5df70cbffa02a6a36e6e18e57b9bd7839cbe13cc20c5b2"} Oct 31 14:00:08.806686 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:08.806282 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" event={"ID":"a5959be5-1f14-4ea5-a4b0-8d6b141052f2","Type":"ContainerStarted","Data":"6ac2669b761080778858060f8a1c4c77e321eba21fb121c666d45efc3c2e9273"} Oct 31 14:00:08.806686 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:08.806506 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" Oct 31 14:00:08.832285 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:08.832230 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" podStartSLOduration=1.832215602 podStartE2EDuration="1.832215602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:08.827535883 +0000 UTC m=+3376.973279717" watchObservedRunningTime="2026-10-31 14:00:08.832215602 +0000 UTC m=+3376.977959429" Oct 31 14:00:13.031909 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:13.031871 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf"] Oct 31 14:00:13.039583 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:13.039509 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp"] Oct 31 14:00:13.044509 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:13.044485 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk"] Oct 31 14:00:13.048554 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:13.048529 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-378cf721-rkrhf"] Oct 31 14:00:13.052460 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:13.052442 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe48eff7-nlmrp"] Oct 31 14:00:13.056211 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:13.056191 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b082606e-pkfjk"] Oct 31 14:00:14.005438 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:14.005408 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40354b49-c2f2-4d25-a9e7-8be1c75c9417" path="/var/lib/kubelet/pods/40354b49-c2f2-4d25-a9e7-8be1c75c9417/volumes" Oct 31 14:00:14.006045 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:14.006029 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7b3051f-13ab-441f-902b-cce0dc8de2a8" path="/var/lib/kubelet/pods/c7b3051f-13ab-441f-902b-cce0dc8de2a8/volumes" Oct 31 14:00:14.006498 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:14.006482 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0009956-628d-4778-8ab0-6c6aa192e10d" path="/var/lib/kubelet/pods/e0009956-628d-4778-8ab0-6c6aa192e10d/volumes" Oct 31 14:00:14.694924 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:14.694887 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2"] Oct 31 14:00:14.787013 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:14.786980 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf"] Oct 31 14:00:14.793247 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:14.793218 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9"] Oct 31 14:00:14.793481 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:14.793458 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" podUID="2928fcdc-f036-4614-a942-c79fc09d9227" containerName="payload-pre-processing" containerID="cri-o://79f7c435d51988369ea7683d887969bbd24f798c23aa6ff5029954a2c253be69" gracePeriod=30 Oct 31 14:00:14.894345 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:14.894311 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr"] Oct 31 14:00:14.905733 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:14.905693 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" podUID="51e61ea6-8c12-4113-a371-a245526ed535" containerName="payload-processing" containerID="cri-o://4112a9c1060111b4b955dc24cf84f8c0f8559afcac15ac1571d689e158a9b662" gracePeriod=30 Oct 31 14:00:14.908844 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:14.908704 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5af6eb-wdvgr"] Oct 31 14:00:14.996362 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:14.996290 5532 scope.go:122] "RemoveContainer" containerID="46b57a2a4cc81532fa49c479aefb1686dc13fe44b4f77a9b35c5f80e59c249f9" Oct 31 14:00:14.996509 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:14.996410 5532 scope.go:122] "RemoveContainer" containerID="bb3471275137564ffb0c949fe3d67c9e9e08dacb93c1f470c3def864f000d867" Oct 31 14:00:15.014695 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.014667 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" Oct 31 14:00:15.269434 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.269151 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-kube-api-access-2x9j7\" (UniqueName: \"kubernetes.io/projected/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-kube-api-access-2x9j7\") pod \"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed\" (UID: \"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed\") " Oct 31 14:00:15.269637 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.269517 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-maas-api-service-ca\") pod \"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed\" (UID: \"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed\") " Oct 31 14:00:15.271029 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.270934 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-maas-api-service-ca" pod "9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed" (UID: "9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:15.272530 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.272233 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-kube-api-access-2x9j7" pod "9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed" (UID: "9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed"). InnerVolumeSpecName "kube-api-access-2x9j7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:15.272530 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.272403 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:15.272530 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.272425 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x9j7\" (UniqueName: \"kubernetes.io/projected/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed-kube-api-access-2x9j7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:15.290309 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.287631 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" Oct 31 14:00:15.375676 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.375137 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2928fcdc-f036-4614-a942-c79fc09d9227-kube-api-access-k6dlb\" (UniqueName: \"kubernetes.io/projected/2928fcdc-f036-4614-a942-c79fc09d9227-kube-api-access-k6dlb\") pod \"2928fcdc-f036-4614-a942-c79fc09d9227\" (UID: \"2928fcdc-f036-4614-a942-c79fc09d9227\") " Oct 31 14:00:15.375676 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.375182 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2928fcdc-f036-4614-a942-c79fc09d9227-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2928fcdc-f036-4614-a942-c79fc09d9227-plugins-config-volume\") pod \"2928fcdc-f036-4614-a942-c79fc09d9227\" (UID: \"2928fcdc-f036-4614-a942-c79fc09d9227\") " Oct 31 14:00:15.376482 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.376454 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2928fcdc-f036-4614-a942-c79fc09d9227-plugins-config-volume" pod "2928fcdc-f036-4614-a942-c79fc09d9227" (UID: "2928fcdc-f036-4614-a942-c79fc09d9227"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:15.441620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.441270 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7"] Oct 31 14:00:15.442835 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.442794 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2928fcdc-f036-4614-a942-c79fc09d9227-kube-api-access-k6dlb" pod "2928fcdc-f036-4614-a942-c79fc09d9227" (UID: "2928fcdc-f036-4614-a942-c79fc09d9227"). InnerVolumeSpecName "kube-api-access-k6dlb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:15.448548 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.448519 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a81979-bcxr7"] Oct 31 14:00:15.476427 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.476401 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" Oct 31 14:00:15.477382 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.477356 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6dlb\" (UniqueName: \"kubernetes.io/projected/2928fcdc-f036-4614-a942-c79fc09d9227-kube-api-access-k6dlb\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:15.477489 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.477383 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2928fcdc-f036-4614-a942-c79fc09d9227-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:15.578288 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.578250 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51e61ea6-8c12-4113-a371-a245526ed535-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e61ea6-8c12-4113-a371-a245526ed535-plugins-config-volume\") pod \"51e61ea6-8c12-4113-a371-a245526ed535\" (UID: \"51e61ea6-8c12-4113-a371-a245526ed535\") " Oct 31 14:00:15.578462 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.578341 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51e61ea6-8c12-4113-a371-a245526ed535-kube-api-access-jd4wq\" (UniqueName: \"kubernetes.io/projected/51e61ea6-8c12-4113-a371-a245526ed535-kube-api-access-jd4wq\") pod \"51e61ea6-8c12-4113-a371-a245526ed535\" (UID: \"51e61ea6-8c12-4113-a371-a245526ed535\") " Oct 31 14:00:15.578629 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.578587 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51e61ea6-8c12-4113-a371-a245526ed535-plugins-config-volume" pod "51e61ea6-8c12-4113-a371-a245526ed535" (UID: "51e61ea6-8c12-4113-a371-a245526ed535"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:15.579046 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.579026 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/51e61ea6-8c12-4113-a371-a245526ed535-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:15.580552 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.580524 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51e61ea6-8c12-4113-a371-a245526ed535-kube-api-access-jd4wq" pod "51e61ea6-8c12-4113-a371-a245526ed535" (UID: "51e61ea6-8c12-4113-a371-a245526ed535"). InnerVolumeSpecName "kube-api-access-jd4wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:15.681038 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.681003 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd4wq\" (UniqueName: \"kubernetes.io/projected/51e61ea6-8c12-4113-a371-a245526ed535-kube-api-access-jd4wq\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:15.916224 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.916184 5532 generic.go:356] "Generic (PLEG): container finished" podID="2928fcdc-f036-4614-a942-c79fc09d9227" containerID="79f7c435d51988369ea7683d887969bbd24f798c23aa6ff5029954a2c253be69" exitCode=0 Oct 31 14:00:15.916678 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.916236 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" Oct 31 14:00:15.916678 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.916263 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" event={"ID":"2928fcdc-f036-4614-a942-c79fc09d9227","Type":"ContainerDied","Data":"79f7c435d51988369ea7683d887969bbd24f798c23aa6ff5029954a2c253be69"} Oct 31 14:00:15.916678 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.916318 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9" event={"ID":"2928fcdc-f036-4614-a942-c79fc09d9227","Type":"ContainerDied","Data":"5a39aca53257087babe6e70bd85c50eaf4d9a3fc378a20292dc20fcc44150ac6"} Oct 31 14:00:15.916678 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.916345 5532 scope.go:122] "RemoveContainer" containerID="79f7c435d51988369ea7683d887969bbd24f798c23aa6ff5029954a2c253be69" Oct 31 14:00:15.918130 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.918105 5532 generic.go:356] "Generic (PLEG): container finished" podID="51e61ea6-8c12-4113-a371-a245526ed535" containerID="4112a9c1060111b4b955dc24cf84f8c0f8559afcac15ac1571d689e158a9b662" exitCode=0 Oct 31 14:00:15.918279 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.918257 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" event={"ID":"51e61ea6-8c12-4113-a371-a245526ed535","Type":"ContainerDied","Data":"4112a9c1060111b4b955dc24cf84f8c0f8559afcac15ac1571d689e158a9b662"} Oct 31 14:00:15.918326 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.918300 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" event={"ID":"51e61ea6-8c12-4113-a371-a245526ed535","Type":"ContainerDied","Data":"409fd8e083a5eebaead86dc864f39e860452d2656e96ac6fe79e8b5a1deedc75"} Oct 31 14:00:15.918367 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.918346 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf" Oct 31 14:00:15.921529 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.921507 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" Oct 31 14:00:15.921686 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.921542 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2" event={"ID":"9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed","Type":"ContainerDied","Data":"ff806ad48357ee97d06a070b01613156c3dc4880342a1822e1834c31ddfbb60c"} Oct 31 14:00:15.924195 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.924170 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" event={"ID":"4a6c45b7-8d74-481a-b429-cc7c65420bdd","Type":"ContainerStarted","Data":"d598c6814b05502f65e395c28ab2b63d55f84a35f7bcb4b3912d1b87348d074a"} Oct 31 14:00:15.928473 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.928449 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" event={"ID":"65744c55-b224-4b04-8705-b26423016c51","Type":"ContainerStarted","Data":"2332109151fc5a047fecf79e1e2cf9a975792a2b4d65f9c38133623506e1ef7b"} Oct 31 14:00:15.930104 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.930086 5532 scope.go:122] "RemoveContainer" containerID="79f7c435d51988369ea7683d887969bbd24f798c23aa6ff5029954a2c253be69" Oct 31 14:00:15.930401 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:15.930379 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f7c435d51988369ea7683d887969bbd24f798c23aa6ff5029954a2c253be69\": container with ID starting with 79f7c435d51988369ea7683d887969bbd24f798c23aa6ff5029954a2c253be69 not found: ID does not exist" containerID="79f7c435d51988369ea7683d887969bbd24f798c23aa6ff5029954a2c253be69" Oct 31 14:00:15.930492 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.930410 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f7c435d51988369ea7683d887969bbd24f798c23aa6ff5029954a2c253be69"} err="failed to get container status \"79f7c435d51988369ea7683d887969bbd24f798c23aa6ff5029954a2c253be69\": rpc error: code = NotFound desc = could not find container \"79f7c435d51988369ea7683d887969bbd24f798c23aa6ff5029954a2c253be69\": container with ID starting with 79f7c435d51988369ea7683d887969bbd24f798c23aa6ff5029954a2c253be69 not found: ID does not exist" Oct 31 14:00:15.930492 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.930426 5532 scope.go:122] "RemoveContainer" containerID="4112a9c1060111b4b955dc24cf84f8c0f8559afcac15ac1571d689e158a9b662" Oct 31 14:00:15.941284 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.941270 5532 scope.go:122] "RemoveContainer" containerID="4112a9c1060111b4b955dc24cf84f8c0f8559afcac15ac1571d689e158a9b662" Oct 31 14:00:15.941549 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:15.941527 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4112a9c1060111b4b955dc24cf84f8c0f8559afcac15ac1571d689e158a9b662\": container with ID starting with 4112a9c1060111b4b955dc24cf84f8c0f8559afcac15ac1571d689e158a9b662 not found: ID does not exist" containerID="4112a9c1060111b4b955dc24cf84f8c0f8559afcac15ac1571d689e158a9b662" Oct 31 14:00:15.941687 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.941554 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4112a9c1060111b4b955dc24cf84f8c0f8559afcac15ac1571d689e158a9b662"} err="failed to get container status \"4112a9c1060111b4b955dc24cf84f8c0f8559afcac15ac1571d689e158a9b662\": rpc error: code = NotFound desc = could not find container \"4112a9c1060111b4b955dc24cf84f8c0f8559afcac15ac1571d689e158a9b662\": container with ID starting with 4112a9c1060111b4b955dc24cf84f8c0f8559afcac15ac1571d689e158a9b662 not found: ID does not exist" Oct 31 14:00:15.941687 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.941568 5532 scope.go:122] "RemoveContainer" containerID="2ab743c2bc4a1724bbd413c2ca9c75238b2e858d3c187b4ddda22ae92597c5ff" Oct 31 14:00:15.972017 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.971991 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9"] Oct 31 14:00:15.978328 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.978300 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a81979-b88bfc495-zc2c9"] Oct 31 14:00:15.996445 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:15.996420 5532 scope.go:122] "RemoveContainer" containerID="01e71f9d6eafb536489845cdf390e88eff83f7b5baf7e202e4cf6e5dc9f7f0cb" Oct 31 14:00:16.009484 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.009450 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2928fcdc-f036-4614-a942-c79fc09d9227" path="/var/lib/kubelet/pods/2928fcdc-f036-4614-a942-c79fc09d9227/volumes" Oct 31 14:00:16.010042 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.010019 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cefbe6c-67a4-4d0b-9246-29a382b4bd96" path="/var/lib/kubelet/pods/4cefbe6c-67a4-4d0b-9246-29a382b4bd96/volumes" Oct 31 14:00:16.010547 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.010525 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d79fc86-7404-4e6f-bd47-14c06efd584d" path="/var/lib/kubelet/pods/5d79fc86-7404-4e6f-bd47-14c06efd584d/volumes" Oct 31 14:00:16.020437 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.018925 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2"] Oct 31 14:00:16.023826 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.023804 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a81979-29890920-gqpz2"] Oct 31 14:00:16.038744 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.038720 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf"] Oct 31 14:00:16.050132 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.050098 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a81979-784f44796d-q4cdf"] Oct 31 14:00:16.938135 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.938103 5532 generic.go:356] "Generic (PLEG): container finished" podID="65744c55-b224-4b04-8705-b26423016c51" containerID="2332109151fc5a047fecf79e1e2cf9a975792a2b4d65f9c38133623506e1ef7b" exitCode=60 Oct 31 14:00:16.938626 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.938178 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" event={"ID":"65744c55-b224-4b04-8705-b26423016c51","Type":"ContainerDied","Data":"2332109151fc5a047fecf79e1e2cf9a975792a2b4d65f9c38133623506e1ef7b"} Oct 31 14:00:16.938626 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.938269 5532 scope.go:122] "RemoveContainer" containerID="46b57a2a4cc81532fa49c479aefb1686dc13fe44b4f77a9b35c5f80e59c249f9" Oct 31 14:00:16.938762 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.938743 5532 scope.go:122] "RemoveContainer" containerID="2332109151fc5a047fecf79e1e2cf9a975792a2b4d65f9c38133623506e1ef7b" Oct 31 14:00:16.939191 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:16.939167 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5_odh-ai-gateway-infra(65744c55-b224-4b04-8705-b26423016c51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" podUID="65744c55-b224-4b04-8705-b26423016c51" Oct 31 14:00:16.942521 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.942493 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" event={"ID":"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef","Type":"ContainerStarted","Data":"8978e50ab44d1deed3f2b0a832dcd52cc46e903348aedd6cf8c1c2d777bdf651"} Oct 31 14:00:16.947042 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.947013 5532 generic.go:356] "Generic (PLEG): container finished" podID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" containerID="d598c6814b05502f65e395c28ab2b63d55f84a35f7bcb4b3912d1b87348d074a" exitCode=60 Oct 31 14:00:16.947141 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.947081 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" event={"ID":"4a6c45b7-8d74-481a-b429-cc7c65420bdd","Type":"ContainerDied","Data":"d598c6814b05502f65e395c28ab2b63d55f84a35f7bcb4b3912d1b87348d074a"} Oct 31 14:00:16.947431 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.947418 5532 scope.go:122] "RemoveContainer" containerID="d598c6814b05502f65e395c28ab2b63d55f84a35f7bcb4b3912d1b87348d074a" Oct 31 14:00:16.947798 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:16.947777 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8_odh-ai-gateway-infra(4a6c45b7-8d74-481a-b429-cc7c65420bdd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" podUID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" Oct 31 14:00:16.952635 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.952589 5532 scope.go:122] "RemoveContainer" containerID="bb3471275137564ffb0c949fe3d67c9e9e08dacb93c1f470c3def864f000d867" Oct 31 14:00:16.960054 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.960034 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8"] Oct 31 14:00:16.996557 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.996525 5532 scope.go:122] "RemoveContainer" containerID="6f5d7a32001cebfb5bd1a7d37c3bda8d640aea25c6f4fc2cd4bdf25ac9f5bbd9" Oct 31 14:00:16.996738 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:16.996660 5532 scope.go:122] "RemoveContainer" containerID="39a0a46a17adcc30b535d0b56a834275f02b36b25553dbf7a0fee76385015539" Oct 31 14:00:17.021218 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:17.021191 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5"] Oct 31 14:00:17.960564 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:17.960535 5532 generic.go:356] "Generic (PLEG): container finished" podID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" containerID="8978e50ab44d1deed3f2b0a832dcd52cc46e903348aedd6cf8c1c2d777bdf651" exitCode=60 Oct 31 14:00:17.960998 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:17.960622 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" event={"ID":"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef","Type":"ContainerDied","Data":"8978e50ab44d1deed3f2b0a832dcd52cc46e903348aedd6cf8c1c2d777bdf651"} Oct 31 14:00:17.960998 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:17.960683 5532 scope.go:122] "RemoveContainer" containerID="01e71f9d6eafb536489845cdf390e88eff83f7b5baf7e202e4cf6e5dc9f7f0cb" Oct 31 14:00:17.961162 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:17.961145 5532 scope.go:122] "RemoveContainer" containerID="8978e50ab44d1deed3f2b0a832dcd52cc46e903348aedd6cf8c1c2d777bdf651" Oct 31 14:00:17.961580 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:17.961556 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv_odh-ai-gateway-infra(f5f8ff8c-ffda-4af4-bc4d-7795c623ddef)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" podUID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" Oct 31 14:00:17.963262 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:17.963238 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" event={"ID":"d2d6b4fa-53be-495d-ba97-f8edf753caa4","Type":"ContainerStarted","Data":"4d2c64b142821876c064813a5d9160c92f13cca0e5379057c84118c6c54d783f"} Oct 31 14:00:17.965439 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:17.965409 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" event={"ID":"d5ab1aa4-e729-443c-a322-1a4ce645be61","Type":"ContainerStarted","Data":"8e0ba6bfadb9f3cceac8a42f5908c1f94213c5387e53a18bb512b735d0d11bd1"} Oct 31 14:00:17.997894 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:17.997851 5532 scope.go:122] "RemoveContainer" containerID="e472bb319ee5df00cc6792883d70ac9192a2cbc9df5fa7a6d56a0cf859a3cf78" Oct 31 14:00:17.998757 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:17.998731 5532 scope.go:122] "RemoveContainer" containerID="3800a9b301317ef03ab85d03481f6bf8480d60566251d817389aa2ffd1eff9e8" Oct 31 14:00:17.999483 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:17.999461 5532 scope.go:122] "RemoveContainer" containerID="de47cbc9f1413a0ce05b3c35262fbe97866ae0b70c91fe1c70e211bdbfb2aab1" Oct 31 14:00:18.016654 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.015607 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51e61ea6-8c12-4113-a371-a245526ed535" path="/var/lib/kubelet/pods/51e61ea6-8c12-4113-a371-a245526ed535/volumes" Oct 31 14:00:18.016654 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.016382 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed" path="/var/lib/kubelet/pods/9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed/volumes" Oct 31 14:00:18.027009 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.025534 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv"] Oct 31 14:00:18.197377 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.197349 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" Oct 31 14:00:18.234329 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.234298 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65744c55-b224-4b04-8705-b26423016c51-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65744c55-b224-4b04-8705-b26423016c51-maas-api-service-ca\") pod \"65744c55-b224-4b04-8705-b26423016c51\" (UID: \"65744c55-b224-4b04-8705-b26423016c51\") " Oct 31 14:00:18.234750 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.234724 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65744c55-b224-4b04-8705-b26423016c51-maas-api-service-ca" pod "65744c55-b224-4b04-8705-b26423016c51" (UID: "65744c55-b224-4b04-8705-b26423016c51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:18.235818 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.235788 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65744c55-b224-4b04-8705-b26423016c51-kube-api-access-ltlz4\" (UniqueName: \"kubernetes.io/projected/65744c55-b224-4b04-8705-b26423016c51-kube-api-access-ltlz4\") pod \"65744c55-b224-4b04-8705-b26423016c51\" (UID: \"65744c55-b224-4b04-8705-b26423016c51\") " Oct 31 14:00:18.236879 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.236827 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65744c55-b224-4b04-8705-b26423016c51-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:18.238124 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.238099 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65744c55-b224-4b04-8705-b26423016c51-kube-api-access-ltlz4" pod "65744c55-b224-4b04-8705-b26423016c51" (UID: "65744c55-b224-4b04-8705-b26423016c51"). InnerVolumeSpecName "kube-api-access-ltlz4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:18.245332 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.245310 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" Oct 31 14:00:18.337632 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.337588 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a6c45b7-8d74-481a-b429-cc7c65420bdd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6c45b7-8d74-481a-b429-cc7c65420bdd-maas-api-service-ca\") pod \"4a6c45b7-8d74-481a-b429-cc7c65420bdd\" (UID: \"4a6c45b7-8d74-481a-b429-cc7c65420bdd\") " Oct 31 14:00:18.337911 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.337889 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a6c45b7-8d74-481a-b429-cc7c65420bdd-kube-api-access-brkgn\" (UniqueName: \"kubernetes.io/projected/4a6c45b7-8d74-481a-b429-cc7c65420bdd-kube-api-access-brkgn\") pod \"4a6c45b7-8d74-481a-b429-cc7c65420bdd\" (UID: \"4a6c45b7-8d74-481a-b429-cc7c65420bdd\") " Oct 31 14:00:18.338019 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.338000 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a6c45b7-8d74-481a-b429-cc7c65420bdd-maas-api-service-ca" pod "4a6c45b7-8d74-481a-b429-cc7c65420bdd" (UID: "4a6c45b7-8d74-481a-b429-cc7c65420bdd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:18.338615 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.338576 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltlz4\" (UniqueName: \"kubernetes.io/projected/65744c55-b224-4b04-8705-b26423016c51-kube-api-access-ltlz4\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:18.338724 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.338639 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a6c45b7-8d74-481a-b429-cc7c65420bdd-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:18.343467 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.343203 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a6c45b7-8d74-481a-b429-cc7c65420bdd-kube-api-access-brkgn" pod "4a6c45b7-8d74-481a-b429-cc7c65420bdd" (UID: "4a6c45b7-8d74-481a-b429-cc7c65420bdd"). InnerVolumeSpecName "kube-api-access-brkgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:18.440487 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.440458 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brkgn\" (UniqueName: \"kubernetes.io/projected/4a6c45b7-8d74-481a-b429-cc7c65420bdd-kube-api-access-brkgn\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:18.991308 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.991266 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" event={"ID":"65744c55-b224-4b04-8705-b26423016c51","Type":"ContainerDied","Data":"ed4f3d423fc1a817d8e2ab18b6448820649683e72a371ef3ae18dc456d55dfe7"} Oct 31 14:00:18.991962 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.991935 5532 scope.go:122] "RemoveContainer" containerID="2332109151fc5a047fecf79e1e2cf9a975792a2b4d65f9c38133623506e1ef7b" Oct 31 14:00:18.992827 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.992807 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5" Oct 31 14:00:18.995796 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:18.995776 5532 scope.go:122] "RemoveContainer" containerID="242fbcb369f73cbde97993a5b5e301fd9191d55b6b6f8eed1e43b83b34de3397" Oct 31 14:00:19.006587 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.006559 5532 generic.go:356] "Generic (PLEG): container finished" podID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" containerID="4d2c64b142821876c064813a5d9160c92f13cca0e5379057c84118c6c54d783f" exitCode=60 Oct 31 14:00:19.006723 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.006695 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" event={"ID":"d2d6b4fa-53be-495d-ba97-f8edf753caa4","Type":"ContainerDied","Data":"4d2c64b142821876c064813a5d9160c92f13cca0e5379057c84118c6c54d783f"} Oct 31 14:00:19.007090 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.007069 5532 scope.go:122] "RemoveContainer" containerID="4d2c64b142821876c064813a5d9160c92f13cca0e5379057c84118c6c54d783f" Oct 31 14:00:19.007387 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:19.007356 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890920-f5wbl_odh-ai-gateway-infra(d2d6b4fa-53be-495d-ba97-f8edf753caa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" podUID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" Oct 31 14:00:19.014768 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.014743 5532 scope.go:122] "RemoveContainer" containerID="6f5d7a32001cebfb5bd1a7d37c3bda8d640aea25c6f4fc2cd4bdf25ac9f5bbd9" Oct 31 14:00:19.024074 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.024045 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl"] Oct 31 14:00:19.025161 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.025133 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" event={"ID":"d5bd142e-0a36-4389-a6f7-da08f0e4c77a","Type":"ContainerStarted","Data":"70d46c85053b929c8c95cb2f48c419ad04f84e3e3f9c787151cb83c095584fb9"} Oct 31 14:00:19.029236 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.029213 5532 generic.go:356] "Generic (PLEG): container finished" podID="d5ab1aa4-e729-443c-a322-1a4ce645be61" containerID="8e0ba6bfadb9f3cceac8a42f5908c1f94213c5387e53a18bb512b735d0d11bd1" exitCode=60 Oct 31 14:00:19.029317 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.029300 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" event={"ID":"d5ab1aa4-e729-443c-a322-1a4ce645be61","Type":"ContainerDied","Data":"8e0ba6bfadb9f3cceac8a42f5908c1f94213c5387e53a18bb512b735d0d11bd1"} Oct 31 14:00:19.029720 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.029700 5532 scope.go:122] "RemoveContainer" containerID="8e0ba6bfadb9f3cceac8a42f5908c1f94213c5387e53a18bb512b735d0d11bd1" Oct 31 14:00:19.030097 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:19.029993 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4_odh-ai-gateway-infra(d5ab1aa4-e729-443c-a322-1a4ce645be61)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" podUID="d5ab1aa4-e729-443c-a322-1a4ce645be61" Oct 31 14:00:19.032160 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.032131 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" event={"ID":"641d472a-fcc8-4c9c-bac0-a721550149a0","Type":"ContainerStarted","Data":"d4622e59900cb399e191ef9d03e8902aa8a83c1e659d332c3ac570091e9f0fe2"} Oct 31 14:00:19.035658 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.035631 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" event={"ID":"4a6c45b7-8d74-481a-b429-cc7c65420bdd","Type":"ContainerDied","Data":"9b9b51041e3577a6a48403c97fc918e26bf08567bcfc2d29182b2178374830da"} Oct 31 14:00:19.035763 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.035743 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8" Oct 31 14:00:19.037996 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.037959 5532 scope.go:122] "RemoveContainer" containerID="39a0a46a17adcc30b535d0b56a834275f02b36b25553dbf7a0fee76385015539" Oct 31 14:00:19.046587 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.046558 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" event={"ID":"27c88326-5db1-4240-bbb5-75b2ed7a72a9","Type":"ContainerStarted","Data":"eb05b33de2dab5408911b87183ca37f2a499a49d31562e0c8dcd14d581bea631"} Oct 31 14:00:19.066642 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.062793 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5"] Oct 31 14:00:19.068004 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.067979 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-14ba5a-29890920-g9jv5"] Oct 31 14:00:19.075111 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.075088 5532 scope.go:122] "RemoveContainer" containerID="d598c6814b05502f65e395c28ab2b63d55f84a35f7bcb4b3912d1b87348d074a" Oct 31 14:00:19.115228 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.115197 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4"] Oct 31 14:00:19.215721 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.215553 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8"] Oct 31 14:00:19.220547 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.220519 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-35d13e-29890920-tn4g8"] Oct 31 14:00:19.478895 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.478864 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" Oct 31 14:00:19.564606 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.564512 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-maas-api-service-ca\") pod \"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef\" (UID: \"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef\") " Oct 31 14:00:19.564784 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.564621 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-kube-api-access-rnlqw\" (UniqueName: \"kubernetes.io/projected/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-kube-api-access-rnlqw\") pod \"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef\" (UID: \"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef\") " Oct 31 14:00:19.564994 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.564967 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-maas-api-service-ca" pod "f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" (UID: "f5f8ff8c-ffda-4af4-bc4d-7795c623ddef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:19.565452 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.565431 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:19.566664 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.566639 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-kube-api-access-rnlqw" pod "f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" (UID: "f5f8ff8c-ffda-4af4-bc4d-7795c623ddef"). InnerVolumeSpecName "kube-api-access-rnlqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:19.666874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.666847 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnlqw\" (UniqueName: \"kubernetes.io/projected/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef-kube-api-access-rnlqw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:19.825748 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:19.825670 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" Oct 31 14:00:20.005429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.005395 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" path="/var/lib/kubelet/pods/4a6c45b7-8d74-481a-b429-cc7c65420bdd/volumes" Oct 31 14:00:20.005979 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.005963 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65744c55-b224-4b04-8705-b26423016c51" path="/var/lib/kubelet/pods/65744c55-b224-4b04-8705-b26423016c51/volumes" Oct 31 14:00:20.062908 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.062868 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" Oct 31 14:00:20.063123 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.062870 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv" event={"ID":"f5f8ff8c-ffda-4af4-bc4d-7795c623ddef","Type":"ContainerDied","Data":"95b7a5b2adb201c1dcfb0d0aef20f8e90d417f62db3d1bde5b3f82ea64c772f6"} Oct 31 14:00:20.063297 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.063164 5532 scope.go:122] "RemoveContainer" containerID="8978e50ab44d1deed3f2b0a832dcd52cc46e903348aedd6cf8c1c2d777bdf651" Oct 31 14:00:20.072202 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.071832 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" event={"ID":"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf","Type":"ContainerStarted","Data":"227d9c17da1d8ea34b3c71a0a31decea6d20938c3874f10ef468683459fced37"} Oct 31 14:00:20.075282 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.075261 5532 generic.go:356] "Generic (PLEG): container finished" podID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" containerID="70d46c85053b929c8c95cb2f48c419ad04f84e3e3f9c787151cb83c095584fb9" exitCode=60 Oct 31 14:00:20.075443 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.075337 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" event={"ID":"d5bd142e-0a36-4389-a6f7-da08f0e4c77a","Type":"ContainerDied","Data":"70d46c85053b929c8c95cb2f48c419ad04f84e3e3f9c787151cb83c095584fb9"} Oct 31 14:00:20.075941 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.075766 5532 scope.go:122] "RemoveContainer" containerID="70d46c85053b929c8c95cb2f48c419ad04f84e3e3f9c787151cb83c095584fb9" Oct 31 14:00:20.076116 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:20.076089 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv_odh-ai-gateway-infra(d5bd142e-0a36-4389-a6f7-da08f0e4c77a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" podUID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" Oct 31 14:00:20.084700 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.084676 5532 scope.go:122] "RemoveContainer" containerID="de47cbc9f1413a0ce05b3c35262fbe97866ae0b70c91fe1c70e211bdbfb2aab1" Oct 31 14:00:20.088046 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.088001 5532 generic.go:356] "Generic (PLEG): container finished" podID="641d472a-fcc8-4c9c-bac0-a721550149a0" containerID="d4622e59900cb399e191ef9d03e8902aa8a83c1e659d332c3ac570091e9f0fe2" exitCode=60 Oct 31 14:00:20.088165 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.088107 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" event={"ID":"641d472a-fcc8-4c9c-bac0-a721550149a0","Type":"ContainerDied","Data":"d4622e59900cb399e191ef9d03e8902aa8a83c1e659d332c3ac570091e9f0fe2"} Oct 31 14:00:20.088658 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.088635 5532 scope.go:122] "RemoveContainer" containerID="d4622e59900cb399e191ef9d03e8902aa8a83c1e659d332c3ac570091e9f0fe2" Oct 31 14:00:20.089064 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:20.089010 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54_odh-ai-gateway-infra(641d472a-fcc8-4c9c-bac0-a721550149a0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" podUID="641d472a-fcc8-4c9c-bac0-a721550149a0" Oct 31 14:00:20.092297 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.092272 5532 generic.go:356] "Generic (PLEG): container finished" podID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" containerID="eb05b33de2dab5408911b87183ca37f2a499a49d31562e0c8dcd14d581bea631" exitCode=60 Oct 31 14:00:20.092395 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.092354 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" event={"ID":"27c88326-5db1-4240-bbb5-75b2ed7a72a9","Type":"ContainerDied","Data":"eb05b33de2dab5408911b87183ca37f2a499a49d31562e0c8dcd14d581bea631"} Oct 31 14:00:20.092777 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.092760 5532 scope.go:122] "RemoveContainer" containerID="eb05b33de2dab5408911b87183ca37f2a499a49d31562e0c8dcd14d581bea631" Oct 31 14:00:20.093050 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:20.093031 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8_odh-ai-gateway-infra(27c88326-5db1-4240-bbb5-75b2ed7a72a9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" podUID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" Oct 31 14:00:20.111339 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.111310 5532 scope.go:122] "RemoveContainer" containerID="3800a9b301317ef03ab85d03481f6bf8480d60566251d817389aa2ffd1eff9e8" Oct 31 14:00:20.133526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.132950 5532 scope.go:122] "RemoveContainer" containerID="e472bb319ee5df00cc6792883d70ac9192a2cbc9df5fa7a6d56a0cf859a3cf78" Oct 31 14:00:20.135070 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.135046 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv"] Oct 31 14:00:20.141683 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.140978 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-974711-29890920-vqcjv"] Oct 31 14:00:20.197227 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.197199 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv"] Oct 31 14:00:20.229646 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.229619 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8"] Oct 31 14:00:20.276192 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.276159 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54"] Oct 31 14:00:20.341817 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.341788 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" Oct 31 14:00:20.375024 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.374996 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" Oct 31 14:00:20.481693 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.481654 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ab1aa4-e729-443c-a322-1a4ce645be61-kube-api-access-xr2vv\" (UniqueName: \"kubernetes.io/projected/d5ab1aa4-e729-443c-a322-1a4ce645be61-kube-api-access-xr2vv\") pod \"d5ab1aa4-e729-443c-a322-1a4ce645be61\" (UID: \"d5ab1aa4-e729-443c-a322-1a4ce645be61\") " Oct 31 14:00:20.481888 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.481723 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ab1aa4-e729-443c-a322-1a4ce645be61-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ab1aa4-e729-443c-a322-1a4ce645be61-maas-api-service-ca\") pod \"d5ab1aa4-e729-443c-a322-1a4ce645be61\" (UID: \"d5ab1aa4-e729-443c-a322-1a4ce645be61\") " Oct 31 14:00:20.481888 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.481764 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2d6b4fa-53be-495d-ba97-f8edf753caa4-kube-api-access-nqzcf\" (UniqueName: \"kubernetes.io/projected/d2d6b4fa-53be-495d-ba97-f8edf753caa4-kube-api-access-nqzcf\") pod \"d2d6b4fa-53be-495d-ba97-f8edf753caa4\" (UID: \"d2d6b4fa-53be-495d-ba97-f8edf753caa4\") " Oct 31 14:00:20.482027 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.481928 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2d6b4fa-53be-495d-ba97-f8edf753caa4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d6b4fa-53be-495d-ba97-f8edf753caa4-maas-api-service-ca\") pod \"d2d6b4fa-53be-495d-ba97-f8edf753caa4\" (UID: \"d2d6b4fa-53be-495d-ba97-f8edf753caa4\") " Oct 31 14:00:20.482137 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.482109 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ab1aa4-e729-443c-a322-1a4ce645be61-maas-api-service-ca" pod "d5ab1aa4-e729-443c-a322-1a4ce645be61" (UID: "d5ab1aa4-e729-443c-a322-1a4ce645be61"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:20.482237 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.482217 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2d6b4fa-53be-495d-ba97-f8edf753caa4-maas-api-service-ca" pod "d2d6b4fa-53be-495d-ba97-f8edf753caa4" (UID: "d2d6b4fa-53be-495d-ba97-f8edf753caa4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:20.482473 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.482455 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ab1aa4-e729-443c-a322-1a4ce645be61-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:20.482516 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.482476 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2d6b4fa-53be-495d-ba97-f8edf753caa4-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:20.483942 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.483919 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ab1aa4-e729-443c-a322-1a4ce645be61-kube-api-access-xr2vv" pod "d5ab1aa4-e729-443c-a322-1a4ce645be61" (UID: "d5ab1aa4-e729-443c-a322-1a4ce645be61"). InnerVolumeSpecName "kube-api-access-xr2vv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:20.484106 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.484090 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2d6b4fa-53be-495d-ba97-f8edf753caa4-kube-api-access-nqzcf" pod "d2d6b4fa-53be-495d-ba97-f8edf753caa4" (UID: "d2d6b4fa-53be-495d-ba97-f8edf753caa4"). InnerVolumeSpecName "kube-api-access-nqzcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:20.584442 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.584402 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr2vv\" (UniqueName: \"kubernetes.io/projected/d5ab1aa4-e729-443c-a322-1a4ce645be61-kube-api-access-xr2vv\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:20.584442 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:20.584439 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqzcf\" (UniqueName: \"kubernetes.io/projected/d2d6b4fa-53be-495d-ba97-f8edf753caa4-kube-api-access-nqzcf\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:21.105764 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.105739 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" Oct 31 14:00:21.105764 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.105743 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl" event={"ID":"d2d6b4fa-53be-495d-ba97-f8edf753caa4","Type":"ContainerDied","Data":"3b9e0f56d4a7e938d4f178c1c0c0998daad8bcdf592f71784ed3ac7e37f6f20c"} Oct 31 14:00:21.106213 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.105844 5532 scope.go:122] "RemoveContainer" containerID="4d2c64b142821876c064813a5d9160c92f13cca0e5379057c84118c6c54d783f" Oct 31 14:00:21.108035 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.108006 5532 generic.go:356] "Generic (PLEG): container finished" podID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" containerID="227d9c17da1d8ea34b3c71a0a31decea6d20938c3874f10ef468683459fced37" exitCode=60 Oct 31 14:00:21.108145 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.108127 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" event={"ID":"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf","Type":"ContainerDied","Data":"227d9c17da1d8ea34b3c71a0a31decea6d20938c3874f10ef468683459fced37"} Oct 31 14:00:21.108643 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.108577 5532 scope.go:122] "RemoveContainer" containerID="227d9c17da1d8ea34b3c71a0a31decea6d20938c3874f10ef468683459fced37" Oct 31 14:00:21.108979 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:21.108954 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr_odh-ai-gateway-infra(a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" podUID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" Oct 31 14:00:21.112022 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.112000 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" event={"ID":"d5ab1aa4-e729-443c-a322-1a4ce645be61","Type":"ContainerDied","Data":"b8e0d7431a37e21d5b210a514961e1bf97ed0510fe1e9f1f2fb11161782c90a0"} Oct 31 14:00:21.112153 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.112008 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4" Oct 31 14:00:21.119906 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.119889 5532 scope.go:122] "RemoveContainer" containerID="242fbcb369f73cbde97993a5b5e301fd9191d55b6b6f8eed1e43b83b34de3397" Oct 31 14:00:21.141887 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.141396 5532 scope.go:122] "RemoveContainer" containerID="8e0ba6bfadb9f3cceac8a42f5908c1f94213c5387e53a18bb512b735d0d11bd1" Oct 31 14:00:21.165010 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.164977 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl"] Oct 31 14:00:21.172644 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.170571 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890920-f5wbl"] Oct 31 14:00:21.215035 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.214765 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr"] Oct 31 14:00:21.273912 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.273881 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4"] Oct 31 14:00:21.299642 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.299369 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-8b2f0a-29890920-5wlb4"] Oct 31 14:00:21.409807 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.409778 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" Oct 31 14:00:21.423329 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.423304 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" Oct 31 14:00:21.446086 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.446058 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" Oct 31 14:00:21.531460 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.531426 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c88326-5db1-4240-bbb5-75b2ed7a72a9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c88326-5db1-4240-bbb5-75b2ed7a72a9-maas-api-service-ca\") pod \"27c88326-5db1-4240-bbb5-75b2ed7a72a9\" (UID: \"27c88326-5db1-4240-bbb5-75b2ed7a72a9\") " Oct 31 14:00:21.531685 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.531505 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-kube-api-access-gzwx5\" (UniqueName: \"kubernetes.io/projected/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-kube-api-access-gzwx5\") pod \"d5bd142e-0a36-4389-a6f7-da08f0e4c77a\" (UID: \"d5bd142e-0a36-4389-a6f7-da08f0e4c77a\") " Oct 31 14:00:21.531685 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.531573 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641d472a-fcc8-4c9c-bac0-a721550149a0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641d472a-fcc8-4c9c-bac0-a721550149a0-maas-api-service-ca\") pod \"641d472a-fcc8-4c9c-bac0-a721550149a0\" (UID: \"641d472a-fcc8-4c9c-bac0-a721550149a0\") " Oct 31 14:00:21.531685 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.531632 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c88326-5db1-4240-bbb5-75b2ed7a72a9-kube-api-access-6m2p7\" (UniqueName: \"kubernetes.io/projected/27c88326-5db1-4240-bbb5-75b2ed7a72a9-kube-api-access-6m2p7\") pod \"27c88326-5db1-4240-bbb5-75b2ed7a72a9\" (UID: \"27c88326-5db1-4240-bbb5-75b2ed7a72a9\") " Oct 31 14:00:21.531899 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.531700 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641d472a-fcc8-4c9c-bac0-a721550149a0-kube-api-access-4w5mq\" (UniqueName: \"kubernetes.io/projected/641d472a-fcc8-4c9c-bac0-a721550149a0-kube-api-access-4w5mq\") pod \"641d472a-fcc8-4c9c-bac0-a721550149a0\" (UID: \"641d472a-fcc8-4c9c-bac0-a721550149a0\") " Oct 31 14:00:21.531899 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.531879 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c88326-5db1-4240-bbb5-75b2ed7a72a9-maas-api-service-ca" pod "27c88326-5db1-4240-bbb5-75b2ed7a72a9" (UID: "27c88326-5db1-4240-bbb5-75b2ed7a72a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:21.532017 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.531911 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-maas-api-service-ca\") pod \"d5bd142e-0a36-4389-a6f7-da08f0e4c77a\" (UID: \"d5bd142e-0a36-4389-a6f7-da08f0e4c77a\") " Oct 31 14:00:21.532403 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.532377 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-maas-api-service-ca" pod "d5bd142e-0a36-4389-a6f7-da08f0e4c77a" (UID: "d5bd142e-0a36-4389-a6f7-da08f0e4c77a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:21.532527 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.532486 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641d472a-fcc8-4c9c-bac0-a721550149a0-maas-api-service-ca" pod "641d472a-fcc8-4c9c-bac0-a721550149a0" (UID: "641d472a-fcc8-4c9c-bac0-a721550149a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:21.532797 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.532772 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/641d472a-fcc8-4c9c-bac0-a721550149a0-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:21.532876 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.532802 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:21.532876 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.532817 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/27c88326-5db1-4240-bbb5-75b2ed7a72a9-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:21.534316 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.534294 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-kube-api-access-gzwx5" pod "d5bd142e-0a36-4389-a6f7-da08f0e4c77a" (UID: "d5bd142e-0a36-4389-a6f7-da08f0e4c77a"). InnerVolumeSpecName "kube-api-access-gzwx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:21.534525 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.534500 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c88326-5db1-4240-bbb5-75b2ed7a72a9-kube-api-access-6m2p7" pod "27c88326-5db1-4240-bbb5-75b2ed7a72a9" (UID: "27c88326-5db1-4240-bbb5-75b2ed7a72a9"). InnerVolumeSpecName "kube-api-access-6m2p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:21.534899 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.534880 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641d472a-fcc8-4c9c-bac0-a721550149a0-kube-api-access-4w5mq" pod "641d472a-fcc8-4c9c-bac0-a721550149a0" (UID: "641d472a-fcc8-4c9c-bac0-a721550149a0"). InnerVolumeSpecName "kube-api-access-4w5mq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:21.633926 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.633850 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w5mq\" (UniqueName: \"kubernetes.io/projected/641d472a-fcc8-4c9c-bac0-a721550149a0-kube-api-access-4w5mq\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:21.633926 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.633879 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzwx5\" (UniqueName: \"kubernetes.io/projected/d5bd142e-0a36-4389-a6f7-da08f0e4c77a-kube-api-access-gzwx5\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:21.633926 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:21.633889 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m2p7\" (UniqueName: \"kubernetes.io/projected/27c88326-5db1-4240-bbb5-75b2ed7a72a9-kube-api-access-6m2p7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:22.006307 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.006272 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" path="/var/lib/kubelet/pods/d2d6b4fa-53be-495d-ba97-f8edf753caa4/volumes" Oct 31 14:00:22.006856 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.006839 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ab1aa4-e729-443c-a322-1a4ce645be61" path="/var/lib/kubelet/pods/d5ab1aa4-e729-443c-a322-1a4ce645be61/volumes" Oct 31 14:00:22.007262 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.007248 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" path="/var/lib/kubelet/pods/f5f8ff8c-ffda-4af4-bc4d-7795c623ddef/volumes" Oct 31 14:00:22.129375 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.129351 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" Oct 31 14:00:22.129864 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.129391 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv" event={"ID":"d5bd142e-0a36-4389-a6f7-da08f0e4c77a","Type":"ContainerDied","Data":"ac41193139ca6a146212d3e5dd83bb9deb024fc9a085658c19a69e604ceb95b8"} Oct 31 14:00:22.129864 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.129460 5532 scope.go:122] "RemoveContainer" containerID="70d46c85053b929c8c95cb2f48c419ad04f84e3e3f9c787151cb83c095584fb9" Oct 31 14:00:22.132884 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.132864 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" Oct 31 14:00:22.133020 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.132883 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54" event={"ID":"641d472a-fcc8-4c9c-bac0-a721550149a0","Type":"ContainerDied","Data":"4a9ddeb5f41179b8231245ba44a5b803ae4c3ea1af8216bf7111fe5a9235a21d"} Oct 31 14:00:22.134639 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.134618 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" Oct 31 14:00:22.134756 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.134647 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8" event={"ID":"27c88326-5db1-4240-bbb5-75b2ed7a72a9","Type":"ContainerDied","Data":"a866f670eb0b893f6f92c7adbcaf164dfa7a7f024a13f41704c0ff4e1acad64b"} Oct 31 14:00:22.142397 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.142380 5532 scope.go:122] "RemoveContainer" containerID="d4622e59900cb399e191ef9d03e8902aa8a83c1e659d332c3ac570091e9f0fe2" Oct 31 14:00:22.153736 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.153714 5532 scope.go:122] "RemoveContainer" containerID="eb05b33de2dab5408911b87183ca37f2a499a49d31562e0c8dcd14d581bea631" Oct 31 14:00:22.168171 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.168139 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8"] Oct 31 14:00:22.173341 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.173315 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-5e8f29-29890920-7cvb8"] Oct 31 14:00:22.186397 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.186372 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54"] Oct 31 14:00:22.192792 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.192768 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-a39e7e-29890920-stj54"] Oct 31 14:00:22.210816 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.210787 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv"] Oct 31 14:00:22.217195 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.217106 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-599291-29890920-tssrv"] Oct 31 14:00:22.300879 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.300853 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" Oct 31 14:00:22.445536 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.445507 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-kube-api-access-wr9q2\" (UniqueName: \"kubernetes.io/projected/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-kube-api-access-wr9q2\") pod \"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf\" (UID: \"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf\") " Oct 31 14:00:22.445756 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.445663 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-maas-api-service-ca\") pod \"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf\" (UID: \"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf\") " Oct 31 14:00:22.446035 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.446005 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-maas-api-service-ca" pod "a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" (UID: "a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:22.446236 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.446220 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:22.447690 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.447668 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-kube-api-access-wr9q2" pod "a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" (UID: "a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf"). InnerVolumeSpecName "kube-api-access-wr9q2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:22.547506 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:22.547413 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr9q2\" (UniqueName: \"kubernetes.io/projected/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf-kube-api-access-wr9q2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:23.147396 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.147367 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" Oct 31 14:00:23.147891 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.147368 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr" event={"ID":"a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf","Type":"ContainerDied","Data":"6b17432cc2e186775464521f03f1bab10527c0f9ed234550c0a44b486c7f84fa"} Oct 31 14:00:23.147891 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.147527 5532 scope.go:122] "RemoveContainer" containerID="227d9c17da1d8ea34b3c71a0a31decea6d20938c3874f10ef468683459fced37" Oct 31 14:00:23.265654 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.263397 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr"] Oct 31 14:00:23.281666 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.276045 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-15dc2f-29890920-mbcqr"] Oct 31 14:00:23.615861 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.615790 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r"] Oct 31 14:00:23.616478 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616463 5532 cpu_manager.go:414] "containerMap: removing container" podUID="641d472a-fcc8-4c9c-bac0-a721550149a0" containerName="cleanup" Oct 31 14:00:23.616517 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616479 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641d472a-fcc8-4c9c-bac0-a721550149a0" containerName="cleanup" Oct 31 14:00:23.616517 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616492 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" containerName="cleanup" Oct 31 14:00:23.616517 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616497 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" containerName="cleanup" Oct 31 14:00:23.616517 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616504 5532 cpu_manager.go:414] "containerMap: removing container" podUID="641d472a-fcc8-4c9c-bac0-a721550149a0" containerName="cleanup" Oct 31 14:00:23.616517 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616509 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641d472a-fcc8-4c9c-bac0-a721550149a0" containerName="cleanup" Oct 31 14:00:23.616517 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616517 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616524 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616534 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616539 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616547 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616551 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616557 5532 cpu_manager.go:414] "containerMap: removing container" podUID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616561 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616569 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616573 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616579 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616584 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616607 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d5ab1aa4-e729-443c-a322-1a4ce645be61" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616612 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ab1aa4-e729-443c-a322-1a4ce645be61" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616622 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d5ab1aa4-e729-443c-a322-1a4ce645be61" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616626 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ab1aa4-e729-443c-a322-1a4ce645be61" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616642 5532 cpu_manager.go:414] "containerMap: removing container" podUID="9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616646 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616654 5532 cpu_manager.go:414] "containerMap: removing container" podUID="65744c55-b224-4b04-8705-b26423016c51" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616658 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65744c55-b224-4b04-8705-b26423016c51" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616669 5532 cpu_manager.go:414] "containerMap: removing container" podUID="65744c55-b224-4b04-8705-b26423016c51" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616673 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65744c55-b224-4b04-8705-b26423016c51" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616681 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616686 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616693 5532 cpu_manager.go:414] "containerMap: removing container" podUID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616698 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616703 5532 cpu_manager.go:414] "containerMap: removing container" podUID="9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616707 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616720 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616724 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616731 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616735 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616744 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2928fcdc-f036-4614-a942-c79fc09d9227" containerName="payload-pre-processing" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616748 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2928fcdc-f036-4614-a942-c79fc09d9227" containerName="payload-pre-processing" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616756 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616760 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" containerName="cleanup" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616775 5532 cpu_manager.go:414] "containerMap: removing container" podUID="51e61ea6-8c12-4113-a371-a245526ed535" containerName="payload-processing" Oct 31 14:00:23.616765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616781 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51e61ea6-8c12-4113-a371-a245526ed535" containerName="payload-processing" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616792 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616797 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616808 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616812 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616920 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616928 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="51e61ea6-8c12-4113-a371-a245526ed535" containerName="payload-processing" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616936 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616942 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616947 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="641d472a-fcc8-4c9c-bac0-a721550149a0" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616952 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616957 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="641d472a-fcc8-4c9c-bac0-a721550149a0" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616962 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616968 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616973 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616978 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616983 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616988 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616992 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ab1aa4-e729-443c-a322-1a4ce645be61" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.616999 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ab1aa4-e729-443c-a322-1a4ce645be61" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.617004 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.617008 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="65744c55-b224-4b04-8705-b26423016c51" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.617014 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="65744c55-b224-4b04-8705-b26423016c51" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.617019 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.617026 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.617032 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2928fcdc-f036-4614-a942-c79fc09d9227" containerName="payload-pre-processing" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.617039 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="65744c55-b224-4b04-8705-b26423016c51" containerName="cleanup" Oct 31 14:00:23.617881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.617044 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" containerName="cleanup" Oct 31 14:00:23.622235 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.622215 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" Oct 31 14:00:23.631131 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.631082 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:00:23.631323 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.631308 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:00:23.634298 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.634274 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r"] Oct 31 14:00:23.776270 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.776215 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r\" (UID: \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" Oct 31 14:00:23.776484 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.776467 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r\" (UID: \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" Oct 31 14:00:23.776539 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.776496 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djs6q\" (UniqueName: \"kubernetes.io/projected/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-kube-api-access-djs6q\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r\" (UID: \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" Oct 31 14:00:23.878497 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.878394 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r\" (UID: \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" Oct 31 14:00:23.878497 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.878452 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djs6q\" (UniqueName: \"kubernetes.io/projected/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-kube-api-access-djs6q\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r\" (UID: \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" Oct 31 14:00:23.878750 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.878610 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r\" (UID: \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" Oct 31 14:00:23.879280 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.879249 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r\" (UID: \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" Oct 31 14:00:23.880899 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.880874 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r\" (UID: \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" Oct 31 14:00:23.892322 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.892300 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djs6q\" (UniqueName: \"kubernetes.io/projected/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-kube-api-access-djs6q\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r\" (UID: \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" Oct 31 14:00:23.939070 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:23.939040 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" Oct 31 14:00:24.012398 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.010630 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" path="/var/lib/kubelet/pods/27c88326-5db1-4240-bbb5-75b2ed7a72a9/volumes" Oct 31 14:00:24.012398 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.011369 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641d472a-fcc8-4c9c-bac0-a721550149a0" path="/var/lib/kubelet/pods/641d472a-fcc8-4c9c-bac0-a721550149a0/volumes" Oct 31 14:00:24.012398 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.012031 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" path="/var/lib/kubelet/pods/a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf/volumes" Oct 31 14:00:24.013851 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.013089 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" path="/var/lib/kubelet/pods/d5bd142e-0a36-4389-a6f7-da08f0e4c77a/volumes" Oct 31 14:00:24.225558 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:00:24.225527 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e7f0c2_a2b3_4588_a7bb_4a81241ec12d.slice/crio-1d5bb8d13335a8b266d052df988e67671d0969107f0062ef186c0aff17d42377 WatchSource:0}: Error finding container 1d5bb8d13335a8b266d052df988e67671d0969107f0062ef186c0aff17d42377: Status 404 returned error can't find the container with id 1d5bb8d13335a8b266d052df988e67671d0969107f0062ef186c0aff17d42377 Oct 31 14:00:24.226628 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.226573 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r"] Oct 31 14:00:24.854094 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.854007 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z"] Oct 31 14:00:24.854924 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.854896 5532 cpu_manager.go:414] "containerMap: removing container" podUID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" containerName="cleanup" Oct 31 14:00:24.854924 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.854919 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" containerName="cleanup" Oct 31 14:00:24.855167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.854943 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" containerName="cleanup" Oct 31 14:00:24.855167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.854951 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" containerName="cleanup" Oct 31 14:00:24.855167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.854994 5532 cpu_manager.go:414] "containerMap: removing container" podUID="641d472a-fcc8-4c9c-bac0-a721550149a0" containerName="cleanup" Oct 31 14:00:24.855167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855003 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641d472a-fcc8-4c9c-bac0-a721550149a0" containerName="cleanup" Oct 31 14:00:24.855167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855024 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" containerName="cleanup" Oct 31 14:00:24.855167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855029 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" containerName="cleanup" Oct 31 14:00:24.855167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855035 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" containerName="cleanup" Oct 31 14:00:24.855167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855043 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a6c45b7-8d74-481a-b429-cc7c65420bdd" containerName="cleanup" Oct 31 14:00:24.855167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855062 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d5ab1aa4-e729-443c-a322-1a4ce645be61" containerName="cleanup" Oct 31 14:00:24.855167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855069 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ab1aa4-e729-443c-a322-1a4ce645be61" containerName="cleanup" Oct 31 14:00:24.855167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855099 5532 cpu_manager.go:414] "containerMap: removing container" podUID="65744c55-b224-4b04-8705-b26423016c51" containerName="cleanup" Oct 31 14:00:24.855167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855107 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65744c55-b224-4b04-8705-b26423016c51" containerName="cleanup" Oct 31 14:00:24.855823 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855260 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e94fac-84f5-4afc-b3ea-9f2c3f7918cf" containerName="cleanup" Oct 31 14:00:24.855823 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855277 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5f8ff8c-ffda-4af4-bc4d-7795c623ddef" containerName="cleanup" Oct 31 14:00:24.855823 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855287 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="641d472a-fcc8-4c9c-bac0-a721550149a0" containerName="cleanup" Oct 31 14:00:24.855823 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855297 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5bd142e-0a36-4389-a6f7-da08f0e4c77a" containerName="cleanup" Oct 31 14:00:24.855823 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855308 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ab1aa4-e729-443c-a322-1a4ce645be61" containerName="cleanup" Oct 31 14:00:24.855823 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855320 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd5a5e9-821f-47b4-b28d-6f73a98ca2ed" containerName="cleanup" Oct 31 14:00:24.855823 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855332 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c88326-5db1-4240-bbb5-75b2ed7a72a9" containerName="cleanup" Oct 31 14:00:24.855823 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.855340 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2d6b4fa-53be-495d-ba97-f8edf753caa4" containerName="cleanup" Oct 31 14:00:24.859694 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.859668 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" Oct 31 14:00:24.867797 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.867768 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z"] Oct 31 14:00:24.998063 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.998028 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nzfz\" (UniqueName: \"kubernetes.io/projected/7ae2bee7-37b1-4b54-8c60-3256a06e478a-kube-api-access-4nzfz\") pod \"maas-api-revoke-sub-c452f5d4-6jh2z\" (UID: \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" Oct 31 14:00:24.998063 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.998069 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c452f5d4-6jh2z\" (UID: \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" Oct 31 14:00:24.998327 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:24.998207 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c452f5d4-6jh2z\" (UID: \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" Oct 31 14:00:25.100241 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:25.100208 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nzfz\" (UniqueName: \"kubernetes.io/projected/7ae2bee7-37b1-4b54-8c60-3256a06e478a-kube-api-access-4nzfz\") pod \"maas-api-revoke-sub-c452f5d4-6jh2z\" (UID: \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" Oct 31 14:00:25.100241 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:25.100251 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c452f5d4-6jh2z\" (UID: \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" Oct 31 14:00:25.100464 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:25.100297 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c452f5d4-6jh2z\" (UID: \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" Oct 31 14:00:25.101063 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:25.101038 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c452f5d4-6jh2z\" (UID: \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" Oct 31 14:00:25.102757 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:25.102731 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c452f5d4-6jh2z\" (UID: \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" Oct 31 14:00:25.114317 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:25.114247 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nzfz\" (UniqueName: \"kubernetes.io/projected/7ae2bee7-37b1-4b54-8c60-3256a06e478a-kube-api-access-4nzfz\") pod \"maas-api-revoke-sub-c452f5d4-6jh2z\" (UID: \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" Oct 31 14:00:25.169150 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:25.169117 5532 generic.go:356] "Generic (PLEG): container finished" podID="93e7f0c2-a2b3-4588-a7bb-4a81241ec12d" containerID="c05d887bb660afca7722a251f59d0c9a1e05633cacacb52be42fcc43e1168136" exitCode=0 Oct 31 14:00:25.169309 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:25.169184 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" event={"ID":"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d","Type":"ContainerDied","Data":"c05d887bb660afca7722a251f59d0c9a1e05633cacacb52be42fcc43e1168136"} Oct 31 14:00:25.169309 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:25.169281 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" event={"ID":"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d","Type":"ContainerStarted","Data":"1d5bb8d13335a8b266d052df988e67671d0969107f0062ef186c0aff17d42377"} Oct 31 14:00:25.192472 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:25.192444 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" Oct 31 14:00:25.511949 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:25.511917 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z"] Oct 31 14:00:25.513879 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:00:25.513849 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ae2bee7_37b1_4b54_8c60_3256a06e478a.slice/crio-505e8f7511110be3d6e62de3a228a39496b8d8788058242399480eb6c8544137 WatchSource:0}: Error finding container 505e8f7511110be3d6e62de3a228a39496b8d8788058242399480eb6c8544137: Status 404 returned error can't find the container with id 505e8f7511110be3d6e62de3a228a39496b8d8788058242399480eb6c8544137 Oct 31 14:00:26.186857 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:26.186804 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" event={"ID":"7ae2bee7-37b1-4b54-8c60-3256a06e478a","Type":"ContainerStarted","Data":"6d708f50d1b6273986aed005b574290159f2cbfa43e29ed3d78a50c52264660c"} Oct 31 14:00:26.187034 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:26.186884 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" event={"ID":"7ae2bee7-37b1-4b54-8c60-3256a06e478a","Type":"ContainerStarted","Data":"505e8f7511110be3d6e62de3a228a39496b8d8788058242399480eb6c8544137"} Oct 31 14:00:26.209410 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:26.209359 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" podStartSLOduration=2.209345969 podStartE2EDuration="2.209345969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:26.208234306 +0000 UTC m=+3394.353978135" watchObservedRunningTime="2026-10-31 14:00:26.209345969 +0000 UTC m=+3394.355089788" Oct 31 14:00:26.348032 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:26.348007 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" Oct 31 14:00:26.527536 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:26.527455 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-kube-api-access-djs6q\" (UniqueName: \"kubernetes.io/projected/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-kube-api-access-djs6q\") pod \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\" (UID: \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\") " Oct 31 14:00:26.527967 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:26.527590 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-openshift-ca\") pod \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\" (UID: \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\") " Oct 31 14:00:26.527967 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:26.527682 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-xks-ca\") pod \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\" (UID: \"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d\") " Oct 31 14:00:26.528055 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:26.527983 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-openshift-ca" pod "93e7f0c2-a2b3-4588-a7bb-4a81241ec12d" (UID: "93e7f0c2-a2b3-4588-a7bb-4a81241ec12d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:26.528321 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:26.528305 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:26.529909 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:26.529884 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-xks-ca" pod "93e7f0c2-a2b3-4588-a7bb-4a81241ec12d" (UID: "93e7f0c2-a2b3-4588-a7bb-4a81241ec12d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:00:26.530056 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:26.530037 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-kube-api-access-djs6q" pod "93e7f0c2-a2b3-4588-a7bb-4a81241ec12d" (UID: "93e7f0c2-a2b3-4588-a7bb-4a81241ec12d"). InnerVolumeSpecName "kube-api-access-djs6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:26.629715 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:26.629680 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djs6q\" (UniqueName: \"kubernetes.io/projected/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-kube-api-access-djs6q\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:26.629715 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:26.629710 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:27.197216 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:27.197188 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" Oct 31 14:00:27.197417 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:27.197226 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r" event={"ID":"93e7f0c2-a2b3-4588-a7bb-4a81241ec12d","Type":"ContainerDied","Data":"1d5bb8d13335a8b266d052df988e67671d0969107f0062ef186c0aff17d42377"} Oct 31 14:00:27.197417 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:27.197273 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d5bb8d13335a8b266d052df988e67671d0969107f0062ef186c0aff17d42377" Oct 31 14:00:27.199174 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:27.199145 5532 generic.go:356] "Generic (PLEG): container finished" podID="7ae2bee7-37b1-4b54-8c60-3256a06e478a" containerID="6d708f50d1b6273986aed005b574290159f2cbfa43e29ed3d78a50c52264660c" exitCode=0 Oct 31 14:00:27.199316 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:27.199212 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" event={"ID":"7ae2bee7-37b1-4b54-8c60-3256a06e478a","Type":"ContainerDied","Data":"6d708f50d1b6273986aed005b574290159f2cbfa43e29ed3d78a50c52264660c"} Oct 31 14:00:28.374929 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:28.374902 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" Oct 31 14:00:28.558769 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:28.558686 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-xks-ca\") pod \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\" (UID: \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\") " Oct 31 14:00:28.558917 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:28.558797 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ae2bee7-37b1-4b54-8c60-3256a06e478a-kube-api-access-4nzfz\" (UniqueName: \"kubernetes.io/projected/7ae2bee7-37b1-4b54-8c60-3256a06e478a-kube-api-access-4nzfz\") pod \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\" (UID: \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\") " Oct 31 14:00:28.558917 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:28.558892 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-openshift-ca\") pod \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\" (UID: \"7ae2bee7-37b1-4b54-8c60-3256a06e478a\") " Oct 31 14:00:28.559261 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:28.559234 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-openshift-ca" pod "7ae2bee7-37b1-4b54-8c60-3256a06e478a" (UID: "7ae2bee7-37b1-4b54-8c60-3256a06e478a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:28.559560 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:28.559543 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:28.561209 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:28.561180 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-xks-ca" pod "7ae2bee7-37b1-4b54-8c60-3256a06e478a" (UID: "7ae2bee7-37b1-4b54-8c60-3256a06e478a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:00:28.561509 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:28.561485 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ae2bee7-37b1-4b54-8c60-3256a06e478a-kube-api-access-4nzfz" pod "7ae2bee7-37b1-4b54-8c60-3256a06e478a" (UID: "7ae2bee7-37b1-4b54-8c60-3256a06e478a"). InnerVolumeSpecName "kube-api-access-4nzfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:28.661568 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:28.661532 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ae2bee7-37b1-4b54-8c60-3256a06e478a-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:28.661568 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:28.661562 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nzfz\" (UniqueName: \"kubernetes.io/projected/7ae2bee7-37b1-4b54-8c60-3256a06e478a-kube-api-access-4nzfz\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:29.032134 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:29.032102 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj"] Oct 31 14:00:29.037943 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:29.037910 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2909b985-d5xfj"] Oct 31 14:00:29.218281 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:29.218242 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" Oct 31 14:00:29.218281 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:29.218276 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z" event={"ID":"7ae2bee7-37b1-4b54-8c60-3256a06e478a","Type":"ContainerDied","Data":"505e8f7511110be3d6e62de3a228a39496b8d8788058242399480eb6c8544137"} Oct 31 14:00:29.218479 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:29.218324 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="505e8f7511110be3d6e62de3a228a39496b8d8788058242399480eb6c8544137" Oct 31 14:00:30.004549 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:30.004509 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43578379-8de5-4e72-9aa6-42880c8af833" path="/var/lib/kubelet/pods/43578379-8de5-4e72-9aa6-42880c8af833/volumes" Oct 31 14:00:31.471004 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:31.470973 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:00:31.762327 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:31.762240 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:00:33.672535 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:33.672501 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m"] Oct 31 14:00:33.688716 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:33.688683 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj"] Oct 31 14:00:35.280980 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.280946 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv"] Oct 31 14:00:35.281741 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.281721 5532 cpu_manager.go:414] "containerMap: removing container" podUID="7ae2bee7-37b1-4b54-8c60-3256a06e478a" containerName="revoke-keys" Oct 31 14:00:35.281741 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.281740 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ae2bee7-37b1-4b54-8c60-3256a06e478a" containerName="revoke-keys" Oct 31 14:00:35.281936 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.281762 5532 cpu_manager.go:414] "containerMap: removing container" podUID="93e7f0c2-a2b3-4588-a7bb-4a81241ec12d" containerName="revoke-keys" Oct 31 14:00:35.281936 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.281767 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e7f0c2-a2b3-4588-a7bb-4a81241ec12d" containerName="revoke-keys" Oct 31 14:00:35.281936 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.281878 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ae2bee7-37b1-4b54-8c60-3256a06e478a" containerName="revoke-keys" Oct 31 14:00:35.281936 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.281889 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e7f0c2-a2b3-4588-a7bb-4a81241ec12d" containerName="revoke-keys" Oct 31 14:00:35.287398 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.287378 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" Oct 31 14:00:35.292799 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.292773 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:00:35.293128 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.293111 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:00:35.296261 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.296234 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv"] Oct 31 14:00:35.395828 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.395790 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab0e8c0d-p22qv\" (UID: \"c1def377-c1f7-475d-a344-1e03e49b0fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" Oct 31 14:00:35.395994 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.395836 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxnfw\" (UniqueName: \"kubernetes.io/projected/c1def377-c1f7-475d-a344-1e03e49b0fc4-kube-api-access-dxnfw\") pod \"maas-api-revoke-sub-ab0e8c0d-p22qv\" (UID: \"c1def377-c1f7-475d-a344-1e03e49b0fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" Oct 31 14:00:35.396255 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.396229 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab0e8c0d-p22qv\" (UID: \"c1def377-c1f7-475d-a344-1e03e49b0fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" Oct 31 14:00:35.506691 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.498649 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab0e8c0d-p22qv\" (UID: \"c1def377-c1f7-475d-a344-1e03e49b0fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" Oct 31 14:00:35.506691 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.498731 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxnfw\" (UniqueName: \"kubernetes.io/projected/c1def377-c1f7-475d-a344-1e03e49b0fc4-kube-api-access-dxnfw\") pod \"maas-api-revoke-sub-ab0e8c0d-p22qv\" (UID: \"c1def377-c1f7-475d-a344-1e03e49b0fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" Oct 31 14:00:35.506691 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.499072 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab0e8c0d-p22qv\" (UID: \"c1def377-c1f7-475d-a344-1e03e49b0fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" Oct 31 14:00:35.506691 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.500479 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ab0e8c0d-p22qv\" (UID: \"c1def377-c1f7-475d-a344-1e03e49b0fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" Oct 31 14:00:35.517390 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.517351 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ab0e8c0d-p22qv\" (UID: \"c1def377-c1f7-475d-a344-1e03e49b0fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" Oct 31 14:00:35.517676 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.517621 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxnfw\" (UniqueName: \"kubernetes.io/projected/c1def377-c1f7-475d-a344-1e03e49b0fc4-kube-api-access-dxnfw\") pod \"maas-api-revoke-sub-ab0e8c0d-p22qv\" (UID: \"c1def377-c1f7-475d-a344-1e03e49b0fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" Oct 31 14:00:35.604586 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.604504 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" Oct 31 14:00:35.899528 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:35.899493 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv"] Oct 31 14:00:35.899727 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:00:35.899690 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1def377_c1f7_475d_a344_1e03e49b0fc4.slice/crio-75f329809f8a7288de9e7493ced43f47aef35e2cf4259e31a86fab49eb64c0f8 WatchSource:0}: Error finding container 75f329809f8a7288de9e7493ced43f47aef35e2cf4259e31a86fab49eb64c0f8: Status 404 returned error can't find the container with id 75f329809f8a7288de9e7493ced43f47aef35e2cf4259e31a86fab49eb64c0f8 Oct 31 14:00:36.365157 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:36.365127 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" event={"ID":"c1def377-c1f7-475d-a344-1e03e49b0fc4","Type":"ContainerStarted","Data":"75f329809f8a7288de9e7493ced43f47aef35e2cf4259e31a86fab49eb64c0f8"} Oct 31 14:00:37.376584 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:37.376492 5532 generic.go:356] "Generic (PLEG): container finished" podID="c1def377-c1f7-475d-a344-1e03e49b0fc4" containerID="4fca01f9392b987f37d79662a2cb58ef131895256ab23a0b43075c8570456c60" exitCode=0 Oct 31 14:00:37.377026 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:37.376574 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" event={"ID":"c1def377-c1f7-475d-a344-1e03e49b0fc4","Type":"ContainerDied","Data":"4fca01f9392b987f37d79662a2cb58ef131895256ab23a0b43075c8570456c60"} Oct 31 14:00:38.559626 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:38.559572 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" Oct 31 14:00:38.648479 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:38.648446 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-xks-ca\") pod \"c1def377-c1f7-475d-a344-1e03e49b0fc4\" (UID: \"c1def377-c1f7-475d-a344-1e03e49b0fc4\") " Oct 31 14:00:38.648666 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:38.648632 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1def377-c1f7-475d-a344-1e03e49b0fc4-kube-api-access-dxnfw\" (UniqueName: \"kubernetes.io/projected/c1def377-c1f7-475d-a344-1e03e49b0fc4-kube-api-access-dxnfw\") pod \"c1def377-c1f7-475d-a344-1e03e49b0fc4\" (UID: \"c1def377-c1f7-475d-a344-1e03e49b0fc4\") " Oct 31 14:00:38.648717 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:38.648705 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-openshift-ca\") pod \"c1def377-c1f7-475d-a344-1e03e49b0fc4\" (UID: \"c1def377-c1f7-475d-a344-1e03e49b0fc4\") " Oct 31 14:00:38.649141 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:38.649108 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-openshift-ca" pod "c1def377-c1f7-475d-a344-1e03e49b0fc4" (UID: "c1def377-c1f7-475d-a344-1e03e49b0fc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:38.649617 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:38.649578 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:38.650574 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:38.650545 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-xks-ca" pod "c1def377-c1f7-475d-a344-1e03e49b0fc4" (UID: "c1def377-c1f7-475d-a344-1e03e49b0fc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:00:38.650915 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:38.650892 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1def377-c1f7-475d-a344-1e03e49b0fc4-kube-api-access-dxnfw" pod "c1def377-c1f7-475d-a344-1e03e49b0fc4" (UID: "c1def377-c1f7-475d-a344-1e03e49b0fc4"). InnerVolumeSpecName "kube-api-access-dxnfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:38.673852 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:38.673805 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" podUID="5c48eb37-c893-4cbf-b24a-9db54e12d699" containerName="payload-processing" containerID="cri-o://f4813f6d5d1910f59369c751044cd17b804f976637e301adb06dd869f1617478" gracePeriod=25 Oct 31 14:00:38.689741 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:38.689697 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" podUID="f0267300-2667-4d7a-8335-fafbc3a4b889" containerName="payload-pre-processing" containerID="cri-o://b0c782305a3441affae92f162767035b51cdb8c6de89f8fb3be83414f3886889" gracePeriod=25 Oct 31 14:00:38.751563 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:38.751527 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxnfw\" (UniqueName: \"kubernetes.io/projected/c1def377-c1f7-475d-a344-1e03e49b0fc4-kube-api-access-dxnfw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:38.751563 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:38.751561 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1def377-c1f7-475d-a344-1e03e49b0fc4-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:39.034758 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.034675 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t"] Oct 31 14:00:39.038419 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.038390 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b5d97f4b-7k27t"] Oct 31 14:00:39.402694 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.402660 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q"] Oct 31 14:00:39.403176 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.403158 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" Oct 31 14:00:39.403472 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.403456 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c1def377-c1f7-475d-a344-1e03e49b0fc4" containerName="revoke-keys" Oct 31 14:00:39.403527 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.403475 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1def377-c1f7-475d-a344-1e03e49b0fc4" containerName="revoke-keys" Oct 31 14:00:39.403587 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.403578 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1def377-c1f7-475d-a344-1e03e49b0fc4" containerName="revoke-keys" Oct 31 14:00:39.407210 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.407186 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv" event={"ID":"c1def377-c1f7-475d-a344-1e03e49b0fc4","Type":"ContainerDied","Data":"75f329809f8a7288de9e7493ced43f47aef35e2cf4259e31a86fab49eb64c0f8"} Oct 31 14:00:39.407311 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.407221 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75f329809f8a7288de9e7493ced43f47aef35e2cf4259e31a86fab49eb64c0f8" Oct 31 14:00:39.407311 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.407293 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" Oct 31 14:00:39.413685 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.412263 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:00:39.413685 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.413443 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:00:39.416358 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.416335 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q"] Oct 31 14:00:39.462684 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.462629 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phdfp\" (UniqueName: \"kubernetes.io/projected/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-kube-api-access-phdfp\") pod \"maas-api-revoke-sub-2fee3c04-c984q\" (UID: \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" Oct 31 14:00:39.462872 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.462794 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fee3c04-c984q\" (UID: \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" Oct 31 14:00:39.462872 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.462851 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fee3c04-c984q\" (UID: \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" Oct 31 14:00:39.564195 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.564162 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phdfp\" (UniqueName: \"kubernetes.io/projected/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-kube-api-access-phdfp\") pod \"maas-api-revoke-sub-2fee3c04-c984q\" (UID: \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" Oct 31 14:00:39.564660 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.564286 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fee3c04-c984q\" (UID: \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" Oct 31 14:00:39.564660 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.564346 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fee3c04-c984q\" (UID: \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" Oct 31 14:00:39.565098 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.565072 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2fee3c04-c984q\" (UID: \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" Oct 31 14:00:39.566956 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.566936 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2fee3c04-c984q\" (UID: \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" Oct 31 14:00:39.577529 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.577503 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phdfp\" (UniqueName: \"kubernetes.io/projected/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-kube-api-access-phdfp\") pod \"maas-api-revoke-sub-2fee3c04-c984q\" (UID: \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" Oct 31 14:00:39.732881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:39.732797 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" Oct 31 14:00:40.005214 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:40.005131 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5335d46-2571-41b5-988e-621350b8ffac" path="/var/lib/kubelet/pods/a5335d46-2571-41b5-988e-621350b8ffac/volumes" Oct 31 14:00:40.029522 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:40.029487 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q"] Oct 31 14:00:40.030039 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:00:40.029915 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda006a12c_ca86_45e0_8b7b_1735b3d02dd4.slice/crio-ec6845bd05cafa8aaec9b6f3c5043fc2e1d39df2ba81fe4c172d311b21562cdf WatchSource:0}: Error finding container ec6845bd05cafa8aaec9b6f3c5043fc2e1d39df2ba81fe4c172d311b21562cdf: Status 404 returned error can't find the container with id ec6845bd05cafa8aaec9b6f3c5043fc2e1d39df2ba81fe4c172d311b21562cdf Oct 31 14:00:40.413670 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:40.413630 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" event={"ID":"a006a12c-ca86-45e0-8b7b-1735b3d02dd4","Type":"ContainerStarted","Data":"ec6845bd05cafa8aaec9b6f3c5043fc2e1d39df2ba81fe4c172d311b21562cdf"} Oct 31 14:00:41.425378 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:41.425341 5532 generic.go:356] "Generic (PLEG): container finished" podID="a006a12c-ca86-45e0-8b7b-1735b3d02dd4" containerID="1e9932aebf1f8fb643883c7fd1526f2e637be1e6094aa362974b7f294245b762" exitCode=0 Oct 31 14:00:41.425875 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:41.425395 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" event={"ID":"a006a12c-ca86-45e0-8b7b-1735b3d02dd4","Type":"ContainerDied","Data":"1e9932aebf1f8fb643883c7fd1526f2e637be1e6094aa362974b7f294245b762"} Oct 31 14:00:42.603989 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:42.603963 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" Oct 31 14:00:42.717258 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:42.717174 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-kube-api-access-phdfp\" (UniqueName: \"kubernetes.io/projected/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-kube-api-access-phdfp\") pod \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\" (UID: \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\") " Oct 31 14:00:42.717444 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:42.717415 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-openshift-ca\") pod \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\" (UID: \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\") " Oct 31 14:00:42.717522 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:42.717443 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-xks-ca\") pod \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\" (UID: \"a006a12c-ca86-45e0-8b7b-1735b3d02dd4\") " Oct 31 14:00:42.717870 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:42.717838 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-openshift-ca" pod "a006a12c-ca86-45e0-8b7b-1735b3d02dd4" (UID: "a006a12c-ca86-45e0-8b7b-1735b3d02dd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:42.718023 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:42.718003 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:42.719490 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:42.719456 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-kube-api-access-phdfp" pod "a006a12c-ca86-45e0-8b7b-1735b3d02dd4" (UID: "a006a12c-ca86-45e0-8b7b-1735b3d02dd4"). InnerVolumeSpecName "kube-api-access-phdfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:42.719632 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:42.719484 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-xks-ca" pod "a006a12c-ca86-45e0-8b7b-1735b3d02dd4" (UID: "a006a12c-ca86-45e0-8b7b-1735b3d02dd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:00:42.822720 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:42.822653 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:42.823064 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:42.823043 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phdfp\" (UniqueName: \"kubernetes.io/projected/a006a12c-ca86-45e0-8b7b-1735b3d02dd4-kube-api-access-phdfp\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:43.443145 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:43.443116 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" Oct 31 14:00:43.443358 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:43.443147 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q" event={"ID":"a006a12c-ca86-45e0-8b7b-1735b3d02dd4","Type":"ContainerDied","Data":"ec6845bd05cafa8aaec9b6f3c5043fc2e1d39df2ba81fe4c172d311b21562cdf"} Oct 31 14:00:43.443358 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:43.443194 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec6845bd05cafa8aaec9b6f3c5043fc2e1d39df2ba81fe4c172d311b21562cdf" Oct 31 14:00:45.501691 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:45.501655 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r"] Oct 31 14:00:45.507039 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:45.507013 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-97927b-xjs5r"] Oct 31 14:00:45.648998 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:45.647061 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" podUID="f0267300-2667-4d7a-8335-fafbc3a4b889" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:00:45.648998 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:45.647471 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" podUID="5c48eb37-c893-4cbf-b24a-9db54e12d699" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:00:46.010000 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:46.009959 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e7f0c2-a2b3-4588-a7bb-4a81241ec12d" path="/var/lib/kubelet/pods/93e7f0c2-a2b3-4588-a7bb-4a81241ec12d/volumes" Oct 31 14:00:46.145820 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:46.145788 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp"] Oct 31 14:00:53.383961 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:53.383281 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4"] Oct 31 14:00:53.610341 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:53.610300 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c"] Oct 31 14:00:53.611433 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:53.611397 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" containerID="cri-o://c4cb2e120b5e9efe4759433e6d2bfcc18dab0045f96ef9075445d6bbe045a6d4" gracePeriod=30 Oct 31 14:00:55.040647 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.032686 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc"] Oct 31 14:00:55.040647 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.032949 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" podUID="bf1757f9-2ed2-447c-a401-ed2feecfa4ca" containerName="istio-proxy" containerID="cri-o://93e6a4a97065671b72b740c635fcabddc77106537fb6a10c347078faf84b6109" gracePeriod=30 Oct 31 14:00:55.549202 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.549169 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4"] Oct 31 14:00:55.549943 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.549927 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a006a12c-ca86-45e0-8b7b-1735b3d02dd4" containerName="revoke-keys" Oct 31 14:00:55.550009 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.549944 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a006a12c-ca86-45e0-8b7b-1735b3d02dd4" containerName="revoke-keys" Oct 31 14:00:55.550065 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.550056 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a006a12c-ca86-45e0-8b7b-1735b3d02dd4" containerName="revoke-keys" Oct 31 14:00:55.553867 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.553849 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" Oct 31 14:00:55.560079 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.560052 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:00:55.561129 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.561104 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:00:55.565472 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.565450 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4"] Oct 31 14:00:55.632880 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.632840 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" podUID="f0267300-2667-4d7a-8335-fafbc3a4b889" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:00:55.633222 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.633195 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" podUID="5c48eb37-c893-4cbf-b24a-9db54e12d699" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:00:55.709006 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.708967 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4\" (UID: \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" Oct 31 14:00:55.709309 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.709259 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x22wq\" (UniqueName: \"kubernetes.io/projected/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-kube-api-access-x22wq\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4\" (UID: \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" Oct 31 14:00:55.709404 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.709343 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4\" (UID: \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" Oct 31 14:00:55.811049 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.810958 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4\" (UID: \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" Oct 31 14:00:55.811210 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.811115 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x22wq\" (UniqueName: \"kubernetes.io/projected/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-kube-api-access-x22wq\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4\" (UID: \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" Oct 31 14:00:55.811210 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.811170 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4\" (UID: \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" Oct 31 14:00:55.811914 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.811891 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4\" (UID: \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" Oct 31 14:00:55.813292 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.813273 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4\" (UID: \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" Oct 31 14:00:55.828242 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.828219 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x22wq\" (UniqueName: \"kubernetes.io/projected/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-kube-api-access-x22wq\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4\" (UID: \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" Oct 31 14:00:55.869797 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:55.869770 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" Oct 31 14:00:56.158419 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:00:56.158389 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7986d3_2b66_4be6_9bb6_16093b0d0dde.slice/crio-b80551bf44b46e4a1c85e3a840299f51cd9b22a6cdf6d926dad74af3df0a9c2e WatchSource:0}: Error finding container b80551bf44b46e4a1c85e3a840299f51cd9b22a6cdf6d926dad74af3df0a9c2e: Status 404 returned error can't find the container with id b80551bf44b46e4a1c85e3a840299f51cd9b22a6cdf6d926dad74af3df0a9c2e Oct 31 14:00:56.159297 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:56.159271 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4"] Oct 31 14:00:56.331969 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:56.331937 5532 scope.go:122] "RemoveContainer" containerID="3819eb07f2479eeccd755bc3d4b9f9eff2518b2f4d003b9fa3c122a706b47db1" Oct 31 14:00:56.343523 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:56.343501 5532 scope.go:122] "RemoveContainer" containerID="b1d66ee333e6fe9a4e73907d1e1822a9c9ba468b9f2606be3e567f527a4d526b" Oct 31 14:00:56.355376 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:56.355353 5532 scope.go:122] "RemoveContainer" containerID="b7c8dd3eb889869571e512ed5979940049d9a2745f4886479b41d85778edcb22" Oct 31 14:00:56.595321 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:56.595289 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" event={"ID":"ee7986d3-2b66-4be6-9bb6-16093b0d0dde","Type":"ContainerStarted","Data":"1b7ee7729977cd8e6bf4071acba25268554e73b5dcd79ca3f015580846144cdd"} Oct 31 14:00:56.595474 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:56.595380 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" event={"ID":"ee7986d3-2b66-4be6-9bb6-16093b0d0dde","Type":"ContainerStarted","Data":"b80551bf44b46e4a1c85e3a840299f51cd9b22a6cdf6d926dad74af3df0a9c2e"} Oct 31 14:00:56.617085 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:56.617040 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" podStartSLOduration=1.6170287 podStartE2EDuration="1.6170287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:00:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:00:56.616503956 +0000 UTC m=+3424.762247785" watchObservedRunningTime="2026-10-31 14:00:56.6170287 +0000 UTC m=+3424.762772528" Oct 31 14:00:57.605921 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:57.605885 5532 generic.go:356] "Generic (PLEG): container finished" podID="ee7986d3-2b66-4be6-9bb6-16093b0d0dde" containerID="1b7ee7729977cd8e6bf4071acba25268554e73b5dcd79ca3f015580846144cdd" exitCode=0 Oct 31 14:00:57.606335 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:57.605959 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" event={"ID":"ee7986d3-2b66-4be6-9bb6-16093b0d0dde","Type":"ContainerDied","Data":"1b7ee7729977cd8e6bf4071acba25268554e73b5dcd79ca3f015580846144cdd"} Oct 31 14:00:58.861184 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:58.860830 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" Oct 31 14:00:58.882972 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:58.879160 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-kube-api-access-x22wq\" (UniqueName: \"kubernetes.io/projected/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-kube-api-access-x22wq\") pod \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\" (UID: \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\") " Oct 31 14:00:58.882972 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:58.879363 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-xks-ca\") pod \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\" (UID: \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\") " Oct 31 14:00:58.882972 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:58.879422 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-openshift-ca\") pod \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\" (UID: \"ee7986d3-2b66-4be6-9bb6-16093b0d0dde\") " Oct 31 14:00:58.882972 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:58.880108 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-openshift-ca" pod "ee7986d3-2b66-4be6-9bb6-16093b0d0dde" (UID: "ee7986d3-2b66-4be6-9bb6-16093b0d0dde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:58.882972 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:58.880701 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:58.883460 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:58.883425 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-xks-ca" pod "ee7986d3-2b66-4be6-9bb6-16093b0d0dde" (UID: "ee7986d3-2b66-4be6-9bb6-16093b0d0dde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:00:58.885093 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:58.885067 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-kube-api-access-x22wq" pod "ee7986d3-2b66-4be6-9bb6-16093b0d0dde" (UID: "ee7986d3-2b66-4be6-9bb6-16093b0d0dde"). InnerVolumeSpecName "kube-api-access-x22wq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:58.968799 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:58.968769 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 14:00:58.982732 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:58.982703 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:58.982857 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:58.982733 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x22wq\" (UniqueName: \"kubernetes.io/projected/ee7986d3-2b66-4be6-9bb6-16093b0d0dde-kube-api-access-x22wq\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:58.993660 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:58.993622 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" Oct 31 14:00:59.062555 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.062508 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" Oct 31 14:00:59.108620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.102905 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/667e09dd-9d9e-431a-8720-c860ef93e504-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667e09dd-9d9e-431a-8720-c860ef93e504-istio-podinfo\") pod \"667e09dd-9d9e-431a-8720-c860ef93e504\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " Oct 31 14:00:59.108620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.103007 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-data\") pod \"667e09dd-9d9e-431a-8720-c860ef93e504\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " Oct 31 14:00:59.108620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.103205 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c48eb37-c893-4cbf-b24a-9db54e12d699-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c48eb37-c893-4cbf-b24a-9db54e12d699-plugins-config-volume\") pod \"5c48eb37-c893-4cbf-b24a-9db54e12d699\" (UID: \"5c48eb37-c893-4cbf-b24a-9db54e12d699\") " Oct 31 14:00:59.108620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.103267 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0267300-2667-4d7a-8335-fafbc3a4b889-kube-api-access-5v5kw\" (UniqueName: \"kubernetes.io/projected/f0267300-2667-4d7a-8335-fafbc3a4b889-kube-api-access-5v5kw\") pod \"f0267300-2667-4d7a-8335-fafbc3a4b889\" (UID: \"f0267300-2667-4d7a-8335-fafbc3a4b889\") " Oct 31 14:00:59.108620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.103348 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-istio-token\" (UniqueName: \"kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-istio-token\") pod \"667e09dd-9d9e-431a-8720-c860ef93e504\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " Oct 31 14:00:59.108620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.103407 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-certs\") pod \"667e09dd-9d9e-431a-8720-c860ef93e504\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " Oct 31 14:00:59.108620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.106403 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-credential-socket\") pod \"667e09dd-9d9e-431a-8720-c860ef93e504\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " Oct 31 14:00:59.108620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.106486 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c48eb37-c893-4cbf-b24a-9db54e12d699-kube-api-access-mn7jj\" (UniqueName: \"kubernetes.io/projected/5c48eb37-c893-4cbf-b24a-9db54e12d699-kube-api-access-mn7jj\") pod \"5c48eb37-c893-4cbf-b24a-9db54e12d699\" (UID: \"5c48eb37-c893-4cbf-b24a-9db54e12d699\") " Oct 31 14:00:59.108620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.106571 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0267300-2667-4d7a-8335-fafbc3a4b889-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0267300-2667-4d7a-8335-fafbc3a4b889-plugins-config-volume\") pod \"f0267300-2667-4d7a-8335-fafbc3a4b889\" (UID: \"f0267300-2667-4d7a-8335-fafbc3a4b889\") " Oct 31 14:00:59.108620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.106649 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-kube-api-access-rvfb7\" (UniqueName: \"kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-kube-api-access-rvfb7\") pod \"667e09dd-9d9e-431a-8720-c860ef93e504\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " Oct 31 14:00:59.108620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.106677 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-socket\") pod \"667e09dd-9d9e-431a-8720-c860ef93e504\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " Oct 31 14:00:59.108620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.106749 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/667e09dd-9d9e-431a-8720-c860ef93e504-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667e09dd-9d9e-431a-8720-c860ef93e504-istiod-ca-cert\") pod \"667e09dd-9d9e-431a-8720-c860ef93e504\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " Oct 31 14:00:59.108620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.106782 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-envoy\") pod \"667e09dd-9d9e-431a-8720-c860ef93e504\" (UID: \"667e09dd-9d9e-431a-8720-c860ef93e504\") " Oct 31 14:00:59.117072 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.115927 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-certs" pod "667e09dd-9d9e-431a-8720-c860ef93e504" (UID: "667e09dd-9d9e-431a-8720-c860ef93e504"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:00:59.128936 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.124455 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/667e09dd-9d9e-431a-8720-c860ef93e504-istio-podinfo" pod "667e09dd-9d9e-431a-8720-c860ef93e504" (UID: "667e09dd-9d9e-431a-8720-c860ef93e504"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:00:59.128936 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.124745 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-data" pod "667e09dd-9d9e-431a-8720-c860ef93e504" (UID: "667e09dd-9d9e-431a-8720-c860ef93e504"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:00:59.128936 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.125054 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c48eb37-c893-4cbf-b24a-9db54e12d699-plugins-config-volume" pod "5c48eb37-c893-4cbf-b24a-9db54e12d699" (UID: "5c48eb37-c893-4cbf-b24a-9db54e12d699"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:59.128936 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.125986 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/667e09dd-9d9e-431a-8720-c860ef93e504-istiod-ca-cert" pod "667e09dd-9d9e-431a-8720-c860ef93e504" (UID: "667e09dd-9d9e-431a-8720-c860ef93e504"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:59.128936 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.126121 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-envoy" pod "667e09dd-9d9e-431a-8720-c860ef93e504" (UID: "667e09dd-9d9e-431a-8720-c860ef93e504"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:00:59.133679 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.130704 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-socket" pod "667e09dd-9d9e-431a-8720-c860ef93e504" (UID: "667e09dd-9d9e-431a-8720-c860ef93e504"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:00:59.133679 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.131484 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0267300-2667-4d7a-8335-fafbc3a4b889-plugins-config-volume" pod "f0267300-2667-4d7a-8335-fafbc3a4b889" (UID: "f0267300-2667-4d7a-8335-fafbc3a4b889"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:00:59.134718 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.134689 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-credential-socket" pod "667e09dd-9d9e-431a-8720-c860ef93e504" (UID: "667e09dd-9d9e-431a-8720-c860ef93e504"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:00:59.139653 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.136783 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c48eb37-c893-4cbf-b24a-9db54e12d699-kube-api-access-mn7jj" pod "5c48eb37-c893-4cbf-b24a-9db54e12d699" (UID: "5c48eb37-c893-4cbf-b24a-9db54e12d699"). InnerVolumeSpecName "kube-api-access-mn7jj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:59.139653 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.136814 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0267300-2667-4d7a-8335-fafbc3a4b889-kube-api-access-5v5kw" pod "f0267300-2667-4d7a-8335-fafbc3a4b889" (UID: "f0267300-2667-4d7a-8335-fafbc3a4b889"). InnerVolumeSpecName "kube-api-access-5v5kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:59.141429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.140752 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-kube-api-access-rvfb7" pod "667e09dd-9d9e-431a-8720-c860ef93e504" (UID: "667e09dd-9d9e-431a-8720-c860ef93e504"). InnerVolumeSpecName "kube-api-access-rvfb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:59.148461 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.148429 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-istio-token" pod "667e09dd-9d9e-431a-8720-c860ef93e504" (UID: "667e09dd-9d9e-431a-8720-c860ef93e504"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:00:59.209445 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.209415 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/667e09dd-9d9e-431a-8720-c860ef93e504-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:59.209445 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.209440 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:59.209445 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.209452 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/667e09dd-9d9e-431a-8720-c860ef93e504-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:59.209705 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.209460 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:59.209705 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.209472 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c48eb37-c893-4cbf-b24a-9db54e12d699-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:59.209705 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.209487 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v5kw\" (UniqueName: \"kubernetes.io/projected/f0267300-2667-4d7a-8335-fafbc3a4b889-kube-api-access-5v5kw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:59.209705 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.209495 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:59.209705 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.209503 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:59.209705 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.209512 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:59.209705 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.209520 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn7jj\" (UniqueName: \"kubernetes.io/projected/5c48eb37-c893-4cbf-b24a-9db54e12d699-kube-api-access-mn7jj\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:59.209705 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.209528 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0267300-2667-4d7a-8335-fafbc3a4b889-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:59.209705 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.209537 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvfb7\" (UniqueName: \"kubernetes.io/projected/667e09dd-9d9e-431a-8720-c860ef93e504-kube-api-access-rvfb7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:59.209705 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.209545 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/667e09dd-9d9e-431a-8720-c860ef93e504-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:00:59.634746 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.634707 5532 generic.go:356] "Generic (PLEG): container finished" podID="f0267300-2667-4d7a-8335-fafbc3a4b889" containerID="b0c782305a3441affae92f162767035b51cdb8c6de89f8fb3be83414f3886889" exitCode=0 Oct 31 14:00:59.634956 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.634772 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" Oct 31 14:00:59.634956 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.634793 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" event={"ID":"f0267300-2667-4d7a-8335-fafbc3a4b889","Type":"ContainerDied","Data":"b0c782305a3441affae92f162767035b51cdb8c6de89f8fb3be83414f3886889"} Oct 31 14:00:59.634956 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.634859 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj" event={"ID":"f0267300-2667-4d7a-8335-fafbc3a4b889","Type":"ContainerDied","Data":"95145a33ed2a2846ae924f446213d6da8e58314addd812ce9ee764c912912406"} Oct 31 14:00:59.634956 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.634890 5532 scope.go:122] "RemoveContainer" containerID="b0c782305a3441affae92f162767035b51cdb8c6de89f8fb3be83414f3886889" Oct 31 14:00:59.636947 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.636733 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" Oct 31 14:00:59.636947 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.636781 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4" event={"ID":"ee7986d3-2b66-4be6-9bb6-16093b0d0dde","Type":"ContainerDied","Data":"b80551bf44b46e4a1c85e3a840299f51cd9b22a6cdf6d926dad74af3df0a9c2e"} Oct 31 14:00:59.636947 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.636821 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b80551bf44b46e4a1c85e3a840299f51cd9b22a6cdf6d926dad74af3df0a9c2e" Oct 31 14:00:59.638404 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.638378 5532 generic.go:356] "Generic (PLEG): container finished" podID="5c48eb37-c893-4cbf-b24a-9db54e12d699" containerID="f4813f6d5d1910f59369c751044cd17b804f976637e301adb06dd869f1617478" exitCode=0 Oct 31 14:00:59.638845 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.638649 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" event={"ID":"5c48eb37-c893-4cbf-b24a-9db54e12d699","Type":"ContainerDied","Data":"f4813f6d5d1910f59369c751044cd17b804f976637e301adb06dd869f1617478"} Oct 31 14:00:59.638845 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.638697 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" event={"ID":"5c48eb37-c893-4cbf-b24a-9db54e12d699","Type":"ContainerDied","Data":"20b54a531e283dc64fe1b607a639e0f1695a3a6b040f47e7e5847293e9285036"} Oct 31 14:00:59.638845 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.638767 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m" Oct 31 14:00:59.646081 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.646057 5532 generic.go:356] "Generic (PLEG): container finished" podID="667e09dd-9d9e-431a-8720-c860ef93e504" containerID="c4cb2e120b5e9efe4759433e6d2bfcc18dab0045f96ef9075445d6bbe045a6d4" exitCode=0 Oct 31 14:00:59.646274 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.646108 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" Oct 31 14:00:59.646510 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.646111 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" event={"ID":"667e09dd-9d9e-431a-8720-c860ef93e504","Type":"ContainerDied","Data":"c4cb2e120b5e9efe4759433e6d2bfcc18dab0045f96ef9075445d6bbe045a6d4"} Oct 31 14:00:59.646755 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.646480 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c" event={"ID":"667e09dd-9d9e-431a-8720-c860ef93e504","Type":"ContainerDied","Data":"831abc3019db0ec021607ec172c3e302cf13a519347a2721f988831015656d32"} Oct 31 14:00:59.670856 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.670828 5532 scope.go:122] "RemoveContainer" containerID="b0c782305a3441affae92f162767035b51cdb8c6de89f8fb3be83414f3886889" Oct 31 14:00:59.671204 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:59.671182 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0c782305a3441affae92f162767035b51cdb8c6de89f8fb3be83414f3886889\": container with ID starting with b0c782305a3441affae92f162767035b51cdb8c6de89f8fb3be83414f3886889 not found: ID does not exist" containerID="b0c782305a3441affae92f162767035b51cdb8c6de89f8fb3be83414f3886889" Oct 31 14:00:59.671299 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.671210 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0c782305a3441affae92f162767035b51cdb8c6de89f8fb3be83414f3886889"} err="failed to get container status \"b0c782305a3441affae92f162767035b51cdb8c6de89f8fb3be83414f3886889\": rpc error: code = NotFound desc = could not find container \"b0c782305a3441affae92f162767035b51cdb8c6de89f8fb3be83414f3886889\": container with ID starting with b0c782305a3441affae92f162767035b51cdb8c6de89f8fb3be83414f3886889 not found: ID does not exist" Oct 31 14:00:59.671299 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.671227 5532 scope.go:122] "RemoveContainer" containerID="f4813f6d5d1910f59369c751044cd17b804f976637e301adb06dd869f1617478" Oct 31 14:00:59.682842 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.682811 5532 scope.go:122] "RemoveContainer" containerID="f4813f6d5d1910f59369c751044cd17b804f976637e301adb06dd869f1617478" Oct 31 14:00:59.683077 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:59.683056 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4813f6d5d1910f59369c751044cd17b804f976637e301adb06dd869f1617478\": container with ID starting with f4813f6d5d1910f59369c751044cd17b804f976637e301adb06dd869f1617478 not found: ID does not exist" containerID="f4813f6d5d1910f59369c751044cd17b804f976637e301adb06dd869f1617478" Oct 31 14:00:59.683126 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.683091 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4813f6d5d1910f59369c751044cd17b804f976637e301adb06dd869f1617478"} err="failed to get container status \"f4813f6d5d1910f59369c751044cd17b804f976637e301adb06dd869f1617478\": rpc error: code = NotFound desc = could not find container \"f4813f6d5d1910f59369c751044cd17b804f976637e301adb06dd869f1617478\": container with ID starting with f4813f6d5d1910f59369c751044cd17b804f976637e301adb06dd869f1617478 not found: ID does not exist" Oct 31 14:00:59.683126 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.683107 5532 scope.go:122] "RemoveContainer" containerID="c4cb2e120b5e9efe4759433e6d2bfcc18dab0045f96ef9075445d6bbe045a6d4" Oct 31 14:00:59.690554 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.690530 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m"] Oct 31 14:00:59.695537 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.695519 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-97927b-5f6bc8df95-m5x5m"] Oct 31 14:00:59.695668 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.695552 5532 scope.go:122] "RemoveContainer" containerID="83fd679e0d8219e8032d6806dedad415d23adaf87642a15d780678cd8cd4a694" Oct 31 14:00:59.705636 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.705586 5532 scope.go:122] "RemoveContainer" containerID="c4cb2e120b5e9efe4759433e6d2bfcc18dab0045f96ef9075445d6bbe045a6d4" Oct 31 14:00:59.705930 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:59.705911 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4cb2e120b5e9efe4759433e6d2bfcc18dab0045f96ef9075445d6bbe045a6d4\": container with ID starting with c4cb2e120b5e9efe4759433e6d2bfcc18dab0045f96ef9075445d6bbe045a6d4 not found: ID does not exist" containerID="c4cb2e120b5e9efe4759433e6d2bfcc18dab0045f96ef9075445d6bbe045a6d4" Oct 31 14:00:59.706006 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.705936 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4cb2e120b5e9efe4759433e6d2bfcc18dab0045f96ef9075445d6bbe045a6d4"} err="failed to get container status \"c4cb2e120b5e9efe4759433e6d2bfcc18dab0045f96ef9075445d6bbe045a6d4\": rpc error: code = NotFound desc = could not find container \"c4cb2e120b5e9efe4759433e6d2bfcc18dab0045f96ef9075445d6bbe045a6d4\": container with ID starting with c4cb2e120b5e9efe4759433e6d2bfcc18dab0045f96ef9075445d6bbe045a6d4 not found: ID does not exist" Oct 31 14:00:59.706006 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.705953 5532 scope.go:122] "RemoveContainer" containerID="83fd679e0d8219e8032d6806dedad415d23adaf87642a15d780678cd8cd4a694" Oct 31 14:00:59.706245 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:00:59.706231 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83fd679e0d8219e8032d6806dedad415d23adaf87642a15d780678cd8cd4a694\": container with ID starting with 83fd679e0d8219e8032d6806dedad415d23adaf87642a15d780678cd8cd4a694 not found: ID does not exist" containerID="83fd679e0d8219e8032d6806dedad415d23adaf87642a15d780678cd8cd4a694" Oct 31 14:00:59.706297 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.706250 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83fd679e0d8219e8032d6806dedad415d23adaf87642a15d780678cd8cd4a694"} err="failed to get container status \"83fd679e0d8219e8032d6806dedad415d23adaf87642a15d780678cd8cd4a694\": rpc error: code = NotFound desc = could not find container \"83fd679e0d8219e8032d6806dedad415d23adaf87642a15d780678cd8cd4a694\": container with ID starting with 83fd679e0d8219e8032d6806dedad415d23adaf87642a15d780678cd8cd4a694 not found: ID does not exist" Oct 31 14:00:59.709399 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.709380 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj"] Oct 31 14:00:59.717151 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.717133 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-97927b-84644994c9-8krrj"] Oct 31 14:00:59.739716 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.739690 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c"] Oct 31 14:00:59.747384 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:00:59.747358 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-9f9103-openshift-default-cb9db67b7-9s57c"] Oct 31 14:01:00.095836 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.095802 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c48eb37-c893-4cbf-b24a-9db54e12d699" path="/var/lib/kubelet/pods/5c48eb37-c893-4cbf-b24a-9db54e12d699/volumes" Oct 31 14:01:00.098021 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.097953 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" path="/var/lib/kubelet/pods/667e09dd-9d9e-431a-8720-c860ef93e504/volumes" Oct 31 14:01:00.100791 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.100761 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0267300-2667-4d7a-8335-fafbc3a4b889" path="/var/lib/kubelet/pods/f0267300-2667-4d7a-8335-fafbc3a4b889/volumes" Oct 31 14:01:00.334569 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.334502 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 14:01:00.440121 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440082 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-socket\") pod \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " Oct 31 14:01:00.440312 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440206 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-certs\") pod \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " Oct 31 14:01:00.440388 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440364 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-podinfo\") pod \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " Oct 31 14:01:00.440445 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440430 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-data\") pod \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " Oct 31 14:01:00.440495 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440483 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-credential-socket\") pod \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " Oct 31 14:01:00.440544 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440513 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-envoy\") pod \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " Oct 31 14:01:00.440544 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440425 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-socket" pod "bf1757f9-2ed2-447c-a401-ed2feecfa4ca" (UID: "bf1757f9-2ed2-447c-a401-ed2feecfa4ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:00.440688 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440532 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-certs" pod "bf1757f9-2ed2-447c-a401-ed2feecfa4ca" (UID: "bf1757f9-2ed2-447c-a401-ed2feecfa4ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:00.440688 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440552 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istiod-ca-cert\") pod \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " Oct 31 14:01:00.440688 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440642 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-token\" (UniqueName: \"kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-token\") pod \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " Oct 31 14:01:00.440868 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440677 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-data" pod "bf1757f9-2ed2-447c-a401-ed2feecfa4ca" (UID: "bf1757f9-2ed2-447c-a401-ed2feecfa4ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:00.440868 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440704 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-kube-api-access-qck4j\" (UniqueName: \"kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-kube-api-access-qck4j\") pod \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\" (UID: \"bf1757f9-2ed2-447c-a401-ed2feecfa4ca\") " Oct 31 14:01:00.440868 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440741 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-credential-socket" pod "bf1757f9-2ed2-447c-a401-ed2feecfa4ca" (UID: "bf1757f9-2ed2-447c-a401-ed2feecfa4ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:00.441028 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.440922 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istiod-ca-cert" pod "bf1757f9-2ed2-447c-a401-ed2feecfa4ca" (UID: "bf1757f9-2ed2-447c-a401-ed2feecfa4ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:00.441467 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.441439 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:00.441467 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.441467 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:00.441704 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.441477 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:00.441704 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.441486 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:00.441704 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.441495 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:00.442840 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.442812 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-podinfo" pod "bf1757f9-2ed2-447c-a401-ed2feecfa4ca" (UID: "bf1757f9-2ed2-447c-a401-ed2feecfa4ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:01:00.442949 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.442878 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-kube-api-access-qck4j" pod "bf1757f9-2ed2-447c-a401-ed2feecfa4ca" (UID: "bf1757f9-2ed2-447c-a401-ed2feecfa4ca"). InnerVolumeSpecName "kube-api-access-qck4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:00.443196 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.443179 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-token" pod "bf1757f9-2ed2-447c-a401-ed2feecfa4ca" (UID: "bf1757f9-2ed2-447c-a401-ed2feecfa4ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:00.443280 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.443265 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-envoy" pod "bf1757f9-2ed2-447c-a401-ed2feecfa4ca" (UID: "bf1757f9-2ed2-447c-a401-ed2feecfa4ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:00.543227 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.543188 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:00.543227 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.543220 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:00.543227 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.543229 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:00.543523 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.543238 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qck4j\" (UniqueName: \"kubernetes.io/projected/bf1757f9-2ed2-447c-a401-ed2feecfa4ca-kube-api-access-qck4j\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:00.657589 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.657556 5532 generic.go:356] "Generic (PLEG): container finished" podID="bf1757f9-2ed2-447c-a401-ed2feecfa4ca" containerID="93e6a4a97065671b72b740c635fcabddc77106537fb6a10c347078faf84b6109" exitCode=0 Oct 31 14:01:00.657796 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.657640 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" Oct 31 14:01:00.657796 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.657639 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" event={"ID":"bf1757f9-2ed2-447c-a401-ed2feecfa4ca","Type":"ContainerDied","Data":"93e6a4a97065671b72b740c635fcabddc77106537fb6a10c347078faf84b6109"} Oct 31 14:01:00.657906 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.657806 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc" event={"ID":"bf1757f9-2ed2-447c-a401-ed2feecfa4ca","Type":"ContainerDied","Data":"bc75410f8a5d5f620d9b338530338074ffb024faaf5f34c50f2621a17845ce81"} Oct 31 14:01:00.657906 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.657840 5532 scope.go:122] "RemoveContainer" containerID="93e6a4a97065671b72b740c635fcabddc77106537fb6a10c347078faf84b6109" Oct 31 14:01:00.670526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.670509 5532 scope.go:122] "RemoveContainer" containerID="93e6a4a97065671b72b740c635fcabddc77106537fb6a10c347078faf84b6109" Oct 31 14:01:00.670832 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:00.670814 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93e6a4a97065671b72b740c635fcabddc77106537fb6a10c347078faf84b6109\": container with ID starting with 93e6a4a97065671b72b740c635fcabddc77106537fb6a10c347078faf84b6109 not found: ID does not exist" containerID="93e6a4a97065671b72b740c635fcabddc77106537fb6a10c347078faf84b6109" Oct 31 14:01:00.670889 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.670841 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93e6a4a97065671b72b740c635fcabddc77106537fb6a10c347078faf84b6109"} err="failed to get container status \"93e6a4a97065671b72b740c635fcabddc77106537fb6a10c347078faf84b6109\": rpc error: code = NotFound desc = could not find container \"93e6a4a97065671b72b740c635fcabddc77106537fb6a10c347078faf84b6109\": container with ID starting with 93e6a4a97065671b72b740c635fcabddc77106537fb6a10c347078faf84b6109 not found: ID does not exist" Oct 31 14:01:00.685985 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.685962 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc"] Oct 31 14:01:00.691581 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:00.691560 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-97927b-openshift-default-5f4c5b677c-d2kmc"] Oct 31 14:01:01.167215 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.167172 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" podUID="b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea" containerName="main" containerID="cri-o://933644d748b13a2a2f9bcb9808ce183d2ff87e1f242493f93a19e50c962f2a78" gracePeriod=45 Oct 31 14:01:01.221230 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.221199 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" podUID="b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea" containerName="main" probeResult="failure" output="Get \"https://10.132.1.11:8000/ready\": dial tcp 10.132.1.11:8000: connect: connection refused" Oct 31 14:01:01.473948 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.473918 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 14:01:01.558066 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.558033 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tls-certs\") pod \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " Oct 31 14:01:01.558236 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.558091 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-kube-api-access-bxlkp\" (UniqueName: \"kubernetes.io/projected/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-kube-api-access-bxlkp\") pod \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " Oct 31 14:01:01.558236 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.558228 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-home\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-home\") pod \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " Oct 31 14:01:01.558393 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.558337 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tmp-dir\") pod \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " Oct 31 14:01:01.558450 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.558407 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-model-cache\") pod \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " Oct 31 14:01:01.558558 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.558506 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-home" pod "b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea" (UID: "b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:01.558643 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.558541 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-dshm\") pod \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\" (UID: \"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea\") " Oct 31 14:01:01.558643 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.558576 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tmp-dir" pod "b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea" (UID: "b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:01.558776 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.558694 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-model-cache" pod "b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea" (UID: "b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:01.559290 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.559268 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:01.559290 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.559289 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:01.559468 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.559300 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:01.560332 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.560311 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tls-certs" pod "b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea" (UID: "b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:01:01.560449 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.560429 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-kube-api-access-bxlkp" pod "b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea" (UID: "b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea"). InnerVolumeSpecName "kube-api-access-bxlkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:01.560584 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.560568 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-dshm" pod "b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea" (UID: "b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:01.660995 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.660966 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:01.660995 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.660994 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxlkp\" (UniqueName: \"kubernetes.io/projected/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-kube-api-access-bxlkp\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:01.661201 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.661004 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:01.671387 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.671355 5532 generic.go:356] "Generic (PLEG): container finished" podID="b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea" containerID="933644d748b13a2a2f9bcb9808ce183d2ff87e1f242493f93a19e50c962f2a78" exitCode=0 Oct 31 14:01:01.671522 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.671400 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" Oct 31 14:01:01.671522 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.671437 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" event={"ID":"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea","Type":"ContainerDied","Data":"933644d748b13a2a2f9bcb9808ce183d2ff87e1f242493f93a19e50c962f2a78"} Oct 31 14:01:01.671522 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.671478 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp" event={"ID":"b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea","Type":"ContainerDied","Data":"fa33590340f7bbf71af163b7b5188ced2a1a2bace80ebdb8ebb3946b57fafb64"} Oct 31 14:01:01.671522 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.671496 5532 scope.go:122] "RemoveContainer" containerID="933644d748b13a2a2f9bcb9808ce183d2ff87e1f242493f93a19e50c962f2a78" Oct 31 14:01:01.682785 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.682762 5532 scope.go:122] "RemoveContainer" containerID="933644d748b13a2a2f9bcb9808ce183d2ff87e1f242493f93a19e50c962f2a78" Oct 31 14:01:01.683079 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:01.683052 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933644d748b13a2a2f9bcb9808ce183d2ff87e1f242493f93a19e50c962f2a78\": container with ID starting with 933644d748b13a2a2f9bcb9808ce183d2ff87e1f242493f93a19e50c962f2a78 not found: ID does not exist" containerID="933644d748b13a2a2f9bcb9808ce183d2ff87e1f242493f93a19e50c962f2a78" Oct 31 14:01:01.683173 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.683087 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933644d748b13a2a2f9bcb9808ce183d2ff87e1f242493f93a19e50c962f2a78"} err="failed to get container status \"933644d748b13a2a2f9bcb9808ce183d2ff87e1f242493f93a19e50c962f2a78\": rpc error: code = NotFound desc = could not find container \"933644d748b13a2a2f9bcb9808ce183d2ff87e1f242493f93a19e50c962f2a78\": container with ID starting with 933644d748b13a2a2f9bcb9808ce183d2ff87e1f242493f93a19e50c962f2a78 not found: ID does not exist" Oct 31 14:01:01.698395 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.698340 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp"] Oct 31 14:01:01.701861 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:01.701842 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-a39e7e/sub-test-model-a39e7e-kserve-59d9f6bd4f-vjptp"] Oct 31 14:01:01.761659 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:01.761624 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:01:02.004782 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:02.004707 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea" path="/var/lib/kubelet/pods/b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea/volumes" Oct 31 14:01:02.005289 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:02.005274 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf1757f9-2ed2-447c-a401-ed2feecfa4ca" path="/var/lib/kubelet/pods/bf1757f9-2ed2-447c-a401-ed2feecfa4ca/volumes" Oct 31 14:01:03.896672 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:03.890877 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp"] Oct 31 14:01:05.812889 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:05.812851 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh"] Oct 31 14:01:07.989586 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:07.989541 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq"] Oct 31 14:01:07.990229 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:07.989805 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" podUID="5738e7ce-0f1c-416f-b3c5-50719400f057" containerName="istio-proxy" containerID="cri-o://1cfe1acb4e20cb8eaf181878d75b86f9047143d0e81166e1de3741458626b5f4" gracePeriod=30 Oct 31 14:01:08.013537 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.013502 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz"] Oct 31 14:01:08.013720 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.013574 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957"] Oct 31 14:01:08.013848 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.013818 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" podUID="51252269-ba06-454a-9a39-04f1fd3896d8" containerName="istio-proxy" containerID="cri-o://56f71a9f143c23b060c9524ec7ce3b37e9fa73f9d03d5a0dd093980feb9025db" gracePeriod=30 Oct 31 14:01:08.014069 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.014048 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" containerID="cri-o://4c31911ad78912113e42a44975329378a63de2df866fa2a104b694b394a00b60" gracePeriod=30 Oct 31 14:01:08.091772 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.091736 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:01:08.091928 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.091801 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" podUID="5738e7ce-0f1c-416f-b3c5-50719400f057" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:01:08.394869 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.394834 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:01:08.425746 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.425574 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" podUID="61795f0c-b2b7-46a8-95b8-5f132f863cf7" containerName="main" containerID="cri-o://75e5a16d934222f4eede96b40d988dfc26226b9dd414bc09829989b84cc6019b" gracePeriod=45 Oct 31 14:01:08.690271 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.690169 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl"] Oct 31 14:01:08.696169 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696139 5532 cpu_manager.go:414] "containerMap: removing container" podUID="5c48eb37-c893-4cbf-b24a-9db54e12d699" containerName="payload-processing" Oct 31 14:01:08.696169 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696166 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c48eb37-c893-4cbf-b24a-9db54e12d699" containerName="payload-processing" Oct 31 14:01:08.696364 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696248 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea" containerName="main" Oct 31 14:01:08.696364 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696273 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea" containerName="main" Oct 31 14:01:08.696364 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696295 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ee7986d3-2b66-4be6-9bb6-16093b0d0dde" containerName="revoke-keys" Oct 31 14:01:08.696364 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696303 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7986d3-2b66-4be6-9bb6-16093b0d0dde" containerName="revoke-keys" Oct 31 14:01:08.696364 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696324 5532 cpu_manager.go:414] "containerMap: removing container" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" Oct 31 14:01:08.696364 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696345 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" Oct 31 14:01:08.696708 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696378 5532 cpu_manager.go:414] "containerMap: removing container" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" Oct 31 14:01:08.696708 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696385 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" Oct 31 14:01:08.696708 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696425 5532 cpu_manager.go:414] "containerMap: removing container" podUID="bf1757f9-2ed2-447c-a401-ed2feecfa4ca" containerName="istio-proxy" Oct 31 14:01:08.696708 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696435 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1757f9-2ed2-447c-a401-ed2feecfa4ca" containerName="istio-proxy" Oct 31 14:01:08.696708 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696455 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f0267300-2667-4d7a-8335-fafbc3a4b889" containerName="payload-pre-processing" Oct 31 14:01:08.696708 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.696462 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0267300-2667-4d7a-8335-fafbc3a4b889" containerName="payload-pre-processing" Oct 31 14:01:08.697072 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.697050 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c48eb37-c893-4cbf-b24a-9db54e12d699" containerName="payload-processing" Oct 31 14:01:08.697133 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.697088 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9094dd3-ce3e-4ee4-a8e2-8af6350f6dea" containerName="main" Oct 31 14:01:08.697133 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.697114 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7986d3-2b66-4be6-9bb6-16093b0d0dde" containerName="revoke-keys" Oct 31 14:01:08.697241 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.697134 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" Oct 31 14:01:08.697241 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.697150 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="667e09dd-9d9e-431a-8720-c860ef93e504" containerName="istio-proxy" Oct 31 14:01:08.697241 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.697181 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1757f9-2ed2-447c-a401-ed2feecfa4ca" containerName="istio-proxy" Oct 31 14:01:08.697241 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.697198 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0267300-2667-4d7a-8335-fafbc3a4b889" containerName="payload-pre-processing" Oct 31 14:01:08.713228 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.710336 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl"] Oct 31 14:01:08.713228 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.710464 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" Oct 31 14:01:08.721580 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.721555 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:01:08.721916 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.721892 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:01:08.744332 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.744296 5532 generic.go:356] "Generic (PLEG): container finished" podID="61795f0c-b2b7-46a8-95b8-5f132f863cf7" containerID="75e5a16d934222f4eede96b40d988dfc26226b9dd414bc09829989b84cc6019b" exitCode=0 Oct 31 14:01:08.744645 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.744619 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" event={"ID":"61795f0c-b2b7-46a8-95b8-5f132f863cf7","Type":"ContainerDied","Data":"75e5a16d934222f4eede96b40d988dfc26226b9dd414bc09829989b84cc6019b"} Oct 31 14:01:08.744768 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.744670 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" event={"ID":"61795f0c-b2b7-46a8-95b8-5f132f863cf7","Type":"ContainerDied","Data":"c707a9135b028bf9dfbbb23ee3c1a9fc5ccb651283640e2bf6b2d99626e0e8b2"} Oct 31 14:01:08.744768 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.744688 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c707a9135b028bf9dfbbb23ee3c1a9fc5ccb651283640e2bf6b2d99626e0e8b2" Oct 31 14:01:08.761638 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.761580 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 14:01:08.893222 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.893188 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tls-certs\" (UniqueName: \"kubernetes.io/secret/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tls-certs\") pod \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " Oct 31 14:01:08.893392 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.893232 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61795f0c-b2b7-46a8-95b8-5f132f863cf7-kube-api-access-qqvk4\" (UniqueName: \"kubernetes.io/projected/61795f0c-b2b7-46a8-95b8-5f132f863cf7-kube-api-access-qqvk4\") pod \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " Oct 31 14:01:08.893392 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.893258 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-home\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-home\") pod \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " Oct 31 14:01:08.893507 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.893395 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tmp-dir\") pod \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " Oct 31 14:01:08.893647 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.893622 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-home" pod "61795f0c-b2b7-46a8-95b8-5f132f863cf7" (UID: "61795f0c-b2b7-46a8-95b8-5f132f863cf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:08.893733 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.893657 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tmp-dir" pod "61795f0c-b2b7-46a8-95b8-5f132f863cf7" (UID: "61795f0c-b2b7-46a8-95b8-5f132f863cf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:08.893733 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.893722 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-dshm\") pod \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " Oct 31 14:01:08.893864 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.893848 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-model-cache\") pod \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\" (UID: \"61795f0c-b2b7-46a8-95b8-5f132f863cf7\") " Oct 31 14:01:08.894145 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.894121 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-model-cache" pod "61795f0c-b2b7-46a8-95b8-5f132f863cf7" (UID: "61795f0c-b2b7-46a8-95b8-5f132f863cf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:08.894292 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.894266 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79127725-b5df-4400-b000-37bbabd42727-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl\" (UID: \"79127725-b5df-4400-b000-37bbabd42727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" Oct 31 14:01:08.894513 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.894495 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgc6r\" (UniqueName: \"kubernetes.io/projected/79127725-b5df-4400-b000-37bbabd42727-kube-api-access-qgc6r\") pod \"maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl\" (UID: \"79127725-b5df-4400-b000-37bbabd42727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" Oct 31 14:01:08.894694 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.894672 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79127725-b5df-4400-b000-37bbabd42727-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl\" (UID: \"79127725-b5df-4400-b000-37bbabd42727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" Oct 31 14:01:08.894922 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.894903 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:08.894967 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.894929 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:08.894967 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.894945 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:08.898049 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.898024 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61795f0c-b2b7-46a8-95b8-5f132f863cf7-kube-api-access-qqvk4" pod "61795f0c-b2b7-46a8-95b8-5f132f863cf7" (UID: "61795f0c-b2b7-46a8-95b8-5f132f863cf7"). InnerVolumeSpecName "kube-api-access-qqvk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:08.898149 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.898070 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-dshm" pod "61795f0c-b2b7-46a8-95b8-5f132f863cf7" (UID: "61795f0c-b2b7-46a8-95b8-5f132f863cf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:08.898149 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.898079 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tls-certs" pod "61795f0c-b2b7-46a8-95b8-5f132f863cf7" (UID: "61795f0c-b2b7-46a8-95b8-5f132f863cf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:01:08.996112 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.996016 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79127725-b5df-4400-b000-37bbabd42727-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl\" (UID: \"79127725-b5df-4400-b000-37bbabd42727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" Oct 31 14:01:08.996625 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.996241 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79127725-b5df-4400-b000-37bbabd42727-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl\" (UID: \"79127725-b5df-4400-b000-37bbabd42727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" Oct 31 14:01:08.996625 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.996354 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgc6r\" (UniqueName: \"kubernetes.io/projected/79127725-b5df-4400-b000-37bbabd42727-kube-api-access-qgc6r\") pod \"maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl\" (UID: \"79127725-b5df-4400-b000-37bbabd42727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" Oct 31 14:01:08.996625 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.996466 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61795f0c-b2b7-46a8-95b8-5f132f863cf7-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:08.996625 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.996483 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqvk4\" (UniqueName: \"kubernetes.io/projected/61795f0c-b2b7-46a8-95b8-5f132f863cf7-kube-api-access-qqvk4\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:08.996625 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.996500 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61795f0c-b2b7-46a8-95b8-5f132f863cf7-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:08.996949 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.996904 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79127725-b5df-4400-b000-37bbabd42727-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl\" (UID: \"79127725-b5df-4400-b000-37bbabd42727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" Oct 31 14:01:08.998516 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:08.998490 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79127725-b5df-4400-b000-37bbabd42727-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl\" (UID: \"79127725-b5df-4400-b000-37bbabd42727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" Oct 31 14:01:09.011172 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:09.011137 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgc6r\" (UniqueName: \"kubernetes.io/projected/79127725-b5df-4400-b000-37bbabd42727-kube-api-access-qgc6r\") pod \"maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl\" (UID: \"79127725-b5df-4400-b000-37bbabd42727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" Oct 31 14:01:09.064410 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:09.064377 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" Oct 31 14:01:09.368167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:09.368136 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl"] Oct 31 14:01:09.368167 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:01:09.368149 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79127725_b5df_4400_b000_37bbabd42727.slice/crio-433a0702c6d7226c15f9d8c4bf6fcfecdc3a6197217b27c21839a5c5e6a273ec WatchSource:0}: Error finding container 433a0702c6d7226c15f9d8c4bf6fcfecdc3a6197217b27c21839a5c5e6a273ec: Status 404 returned error can't find the container with id 433a0702c6d7226c15f9d8c4bf6fcfecdc3a6197217b27c21839a5c5e6a273ec Oct 31 14:01:09.755053 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:09.755017 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" event={"ID":"79127725-b5df-4400-b000-37bbabd42727","Type":"ContainerStarted","Data":"433a0702c6d7226c15f9d8c4bf6fcfecdc3a6197217b27c21839a5c5e6a273ec"} Oct 31 14:01:09.755250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:09.755165 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4" Oct 31 14:01:09.786012 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:09.785987 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4"] Oct 31 14:01:09.790908 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:09.790883 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-35d13e/sub-test-model-35d13e-kserve-58587ffff5-8s9k4"] Oct 31 14:01:10.004862 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:10.004827 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61795f0c-b2b7-46a8-95b8-5f132f863cf7" path="/var/lib/kubelet/pods/61795f0c-b2b7-46a8-95b8-5f132f863cf7/volumes" Oct 31 14:01:10.536627 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:10.536559 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:01:10.536796 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:10.536665 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" podUID="51252269-ba06-454a-9a39-04f1fd3896d8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:01:10.766572 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:10.766541 5532 generic.go:356] "Generic (PLEG): container finished" podID="79127725-b5df-4400-b000-37bbabd42727" containerID="1fdd33f46a7de9f801a9e43fada789f6bfabeeb20f78223bd125b38412f05d63" exitCode=0 Oct 31 14:01:10.766781 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:10.766584 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" event={"ID":"79127725-b5df-4400-b000-37bbabd42727","Type":"ContainerDied","Data":"1fdd33f46a7de9f801a9e43fada789f6bfabeeb20f78223bd125b38412f05d63"} Oct 31 14:01:10.813373 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:10.813339 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" podUID="b4497eeb-2f9a-4131-b3e4-3fcef824c637" containerName="payload-pre-processing" containerID="cri-o://09e29943cee80b121099b7438491b95e63ede76290297e477a5cf0d99d5882f0" gracePeriod=25 Oct 31 14:01:11.923255 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:11.923228 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" Oct 31 14:01:11.949393 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:11.949358 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79127725-b5df-4400-b000-37bbabd42727-kube-api-access-qgc6r\" (UniqueName: \"kubernetes.io/projected/79127725-b5df-4400-b000-37bbabd42727-kube-api-access-qgc6r\") pod \"79127725-b5df-4400-b000-37bbabd42727\" (UID: \"79127725-b5df-4400-b000-37bbabd42727\") " Oct 31 14:01:11.949570 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:11.949467 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79127725-b5df-4400-b000-37bbabd42727-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79127725-b5df-4400-b000-37bbabd42727-maas-api-xks-ca\") pod \"79127725-b5df-4400-b000-37bbabd42727\" (UID: \"79127725-b5df-4400-b000-37bbabd42727\") " Oct 31 14:01:11.949570 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:11.949524 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79127725-b5df-4400-b000-37bbabd42727-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79127725-b5df-4400-b000-37bbabd42727-maas-api-openshift-ca\") pod \"79127725-b5df-4400-b000-37bbabd42727\" (UID: \"79127725-b5df-4400-b000-37bbabd42727\") " Oct 31 14:01:11.950217 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:11.950184 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79127725-b5df-4400-b000-37bbabd42727-maas-api-openshift-ca" pod "79127725-b5df-4400-b000-37bbabd42727" (UID: "79127725-b5df-4400-b000-37bbabd42727"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:11.950520 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:11.950497 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79127725-b5df-4400-b000-37bbabd42727-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:11.951971 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:11.951919 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79127725-b5df-4400-b000-37bbabd42727-maas-api-xks-ca" pod "79127725-b5df-4400-b000-37bbabd42727" (UID: "79127725-b5df-4400-b000-37bbabd42727"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:01:11.952845 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:11.952818 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79127725-b5df-4400-b000-37bbabd42727-kube-api-access-qgc6r" pod "79127725-b5df-4400-b000-37bbabd42727" (UID: "79127725-b5df-4400-b000-37bbabd42727"). InnerVolumeSpecName "kube-api-access-qgc6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:12.052098 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:12.052010 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgc6r\" (UniqueName: \"kubernetes.io/projected/79127725-b5df-4400-b000-37bbabd42727-kube-api-access-qgc6r\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:12.052098 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:12.052042 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79127725-b5df-4400-b000-37bbabd42727-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:12.785035 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:12.785006 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" Oct 31 14:01:12.785207 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:12.785033 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl" event={"ID":"79127725-b5df-4400-b000-37bbabd42727","Type":"ContainerDied","Data":"433a0702c6d7226c15f9d8c4bf6fcfecdc3a6197217b27c21839a5c5e6a273ec"} Oct 31 14:01:12.785207 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:12.785084 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="433a0702c6d7226c15f9d8c4bf6fcfecdc3a6197217b27c21839a5c5e6a273ec" Oct 31 14:01:12.835723 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:12.835686 5532 patch_prober.go:29] interesting pod/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:01:12.835892 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:12.835743 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:01:13.342811 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.342739 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.372326 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-token\" (UniqueName: \"kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-token\") pod \"5738e7ce-0f1c-416f-b3c5-50719400f057\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.372426 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-kube-api-access-cnjks\" (UniqueName: \"kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-kube-api-access-cnjks\") pod \"5738e7ce-0f1c-416f-b3c5-50719400f057\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.372554 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-data\") pod \"5738e7ce-0f1c-416f-b3c5-50719400f057\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.372662 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-credential-socket\") pod \"5738e7ce-0f1c-416f-b3c5-50719400f057\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.372744 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-socket\") pod \"5738e7ce-0f1c-416f-b3c5-50719400f057\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.372801 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-podinfo\") pod \"5738e7ce-0f1c-416f-b3c5-50719400f057\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.372852 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-certs\") pod \"5738e7ce-0f1c-416f-b3c5-50719400f057\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.372884 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-envoy\") pod \"5738e7ce-0f1c-416f-b3c5-50719400f057\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.372964 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5738e7ce-0f1c-416f-b3c5-50719400f057-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5738e7ce-0f1c-416f-b3c5-50719400f057-istiod-ca-cert\") pod \"5738e7ce-0f1c-416f-b3c5-50719400f057\" (UID: \"5738e7ce-0f1c-416f-b3c5-50719400f057\") " Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.372979 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-data" pod "5738e7ce-0f1c-416f-b3c5-50719400f057" (UID: "5738e7ce-0f1c-416f-b3c5-50719400f057"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.373711 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-credential-socket" pod "5738e7ce-0f1c-416f-b3c5-50719400f057" (UID: "5738e7ce-0f1c-416f-b3c5-50719400f057"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.373731 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.373816 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-certs" pod "5738e7ce-0f1c-416f-b3c5-50719400f057" (UID: "5738e7ce-0f1c-416f-b3c5-50719400f057"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.374009 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-socket" pod "5738e7ce-0f1c-416f-b3c5-50719400f057" (UID: "5738e7ce-0f1c-416f-b3c5-50719400f057"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.375208 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.374063 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5738e7ce-0f1c-416f-b3c5-50719400f057-istiod-ca-cert" pod "5738e7ce-0f1c-416f-b3c5-50719400f057" (UID: "5738e7ce-0f1c-416f-b3c5-50719400f057"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:13.429848 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.429811 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-podinfo" pod "5738e7ce-0f1c-416f-b3c5-50719400f057" (UID: "5738e7ce-0f1c-416f-b3c5-50719400f057"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:01:13.430014 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.429876 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-token" pod "5738e7ce-0f1c-416f-b3c5-50719400f057" (UID: "5738e7ce-0f1c-416f-b3c5-50719400f057"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:13.430175 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.430143 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-kube-api-access-cnjks" pod "5738e7ce-0f1c-416f-b3c5-50719400f057" (UID: "5738e7ce-0f1c-416f-b3c5-50719400f057"). InnerVolumeSpecName "kube-api-access-cnjks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:13.431637 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.431576 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-envoy" pod "5738e7ce-0f1c-416f-b3c5-50719400f057" (UID: "5738e7ce-0f1c-416f-b3c5-50719400f057"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.433004 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.432983 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 14:01:13.435111 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.435093 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 14:01:13.474909 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.474843 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-kube-api-access-zss25\" (UniqueName: \"kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-kube-api-access-zss25\") pod \"51252269-ba06-454a-9a39-04f1fd3896d8\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " Oct 31 14:01:13.475112 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.474984 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ee59769-d44a-4690-b5b0-aeecb07beff2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee59769-d44a-4690-b5b0-aeecb07beff2-istiod-ca-cert\") pod \"7ee59769-d44a-4690-b5b0-aeecb07beff2\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " Oct 31 14:01:13.475112 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475026 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-socket\") pod \"7ee59769-d44a-4690-b5b0-aeecb07beff2\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " Oct 31 14:01:13.475112 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475064 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-envoy\") pod \"51252269-ba06-454a-9a39-04f1fd3896d8\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " Oct 31 14:01:13.475112 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475091 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-kube-api-access-xgskr\" (UniqueName: \"kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-kube-api-access-xgskr\") pod \"7ee59769-d44a-4690-b5b0-aeecb07beff2\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " Oct 31 14:01:13.475112 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475113 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-credential-socket\") pod \"51252269-ba06-454a-9a39-04f1fd3896d8\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " Oct 31 14:01:13.475442 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475139 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51252269-ba06-454a-9a39-04f1fd3896d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51252269-ba06-454a-9a39-04f1fd3896d8-istio-podinfo\") pod \"51252269-ba06-454a-9a39-04f1fd3896d8\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " Oct 31 14:01:13.475442 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475183 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-podinfo\") pod \"7ee59769-d44a-4690-b5b0-aeecb07beff2\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " Oct 31 14:01:13.475442 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475226 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-certs\") pod \"7ee59769-d44a-4690-b5b0-aeecb07beff2\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " Oct 31 14:01:13.475442 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475282 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51252269-ba06-454a-9a39-04f1fd3896d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51252269-ba06-454a-9a39-04f1fd3896d8-istiod-ca-cert\") pod \"51252269-ba06-454a-9a39-04f1fd3896d8\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " Oct 31 14:01:13.475442 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475391 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-envoy\") pod \"7ee59769-d44a-4690-b5b0-aeecb07beff2\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " Oct 31 14:01:13.475442 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475429 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-socket\") pod \"51252269-ba06-454a-9a39-04f1fd3896d8\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " Oct 31 14:01:13.475804 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475474 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-credential-socket" pod "51252269-ba06-454a-9a39-04f1fd3896d8" (UID: "51252269-ba06-454a-9a39-04f1fd3896d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.475804 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475484 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-data\") pod \"7ee59769-d44a-4690-b5b0-aeecb07beff2\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " Oct 31 14:01:13.475804 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475574 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-certs\") pod \"51252269-ba06-454a-9a39-04f1fd3896d8\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " Oct 31 14:01:13.475804 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475681 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-data\") pod \"51252269-ba06-454a-9a39-04f1fd3896d8\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " Oct 31 14:01:13.475804 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475713 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-socket" pod "7ee59769-d44a-4690-b5b0-aeecb07beff2" (UID: "7ee59769-d44a-4690-b5b0-aeecb07beff2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.475804 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475729 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-data" pod "7ee59769-d44a-4690-b5b0-aeecb07beff2" (UID: "7ee59769-d44a-4690-b5b0-aeecb07beff2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.475804 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475731 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-token\" (UniqueName: \"kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-token\") pod \"7ee59769-d44a-4690-b5b0-aeecb07beff2\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " Oct 31 14:01:13.476198 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475838 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-credential-socket\") pod \"7ee59769-d44a-4690-b5b0-aeecb07beff2\" (UID: \"7ee59769-d44a-4690-b5b0-aeecb07beff2\") " Oct 31 14:01:13.476198 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475912 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-istio-token\" (UniqueName: \"kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-istio-token\") pod \"51252269-ba06-454a-9a39-04f1fd3896d8\" (UID: \"51252269-ba06-454a-9a39-04f1fd3896d8\") " Oct 31 14:01:13.476198 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.475956 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-certs" pod "51252269-ba06-454a-9a39-04f1fd3896d8" (UID: "51252269-ba06-454a-9a39-04f1fd3896d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.476456 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476429 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-data" pod "51252269-ba06-454a-9a39-04f1fd3896d8" (UID: "51252269-ba06-454a-9a39-04f1fd3896d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.477200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476803 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.477200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476833 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.477200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476849 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.477200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476862 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.477200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476877 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5738e7ce-0f1c-416f-b3c5-50719400f057-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.477200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476891 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.477200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476906 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.477200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476920 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.477200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476934 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnjks\" (UniqueName: \"kubernetes.io/projected/5738e7ce-0f1c-416f-b3c5-50719400f057-kube-api-access-cnjks\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.477200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476948 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.477200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476964 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.477200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476977 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.477200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.476991 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5738e7ce-0f1c-416f-b3c5-50719400f057-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.478232 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.478200 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-envoy" pod "51252269-ba06-454a-9a39-04f1fd3896d8" (UID: "51252269-ba06-454a-9a39-04f1fd3896d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.478474 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.478450 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-socket" pod "51252269-ba06-454a-9a39-04f1fd3896d8" (UID: "51252269-ba06-454a-9a39-04f1fd3896d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.478907 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.478883 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51252269-ba06-454a-9a39-04f1fd3896d8-istiod-ca-cert" pod "51252269-ba06-454a-9a39-04f1fd3896d8" (UID: "51252269-ba06-454a-9a39-04f1fd3896d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:13.479174 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.479149 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-istio-token" pod "51252269-ba06-454a-9a39-04f1fd3896d8" (UID: "51252269-ba06-454a-9a39-04f1fd3896d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:13.479250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.479180 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-certs" pod "7ee59769-d44a-4690-b5b0-aeecb07beff2" (UID: "7ee59769-d44a-4690-b5b0-aeecb07beff2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.479389 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.479369 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-credential-socket" pod "7ee59769-d44a-4690-b5b0-aeecb07beff2" (UID: "7ee59769-d44a-4690-b5b0-aeecb07beff2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.479864 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.479837 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ee59769-d44a-4690-b5b0-aeecb07beff2-istiod-ca-cert" pod "7ee59769-d44a-4690-b5b0-aeecb07beff2" (UID: "7ee59769-d44a-4690-b5b0-aeecb07beff2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:13.480145 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.479981 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-kube-api-access-zss25" pod "51252269-ba06-454a-9a39-04f1fd3896d8" (UID: "51252269-ba06-454a-9a39-04f1fd3896d8"). InnerVolumeSpecName "kube-api-access-zss25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:13.480145 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.480112 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-token" pod "7ee59769-d44a-4690-b5b0-aeecb07beff2" (UID: "7ee59769-d44a-4690-b5b0-aeecb07beff2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:13.480395 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.480365 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-podinfo" pod "7ee59769-d44a-4690-b5b0-aeecb07beff2" (UID: "7ee59769-d44a-4690-b5b0-aeecb07beff2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:01:13.481019 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.480995 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-envoy" pod "7ee59769-d44a-4690-b5b0-aeecb07beff2" (UID: "7ee59769-d44a-4690-b5b0-aeecb07beff2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:13.481668 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.481633 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-kube-api-access-xgskr" pod "7ee59769-d44a-4690-b5b0-aeecb07beff2" (UID: "7ee59769-d44a-4690-b5b0-aeecb07beff2"). InnerVolumeSpecName "kube-api-access-xgskr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:13.482145 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.482125 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51252269-ba06-454a-9a39-04f1fd3896d8-istio-podinfo" pod "51252269-ba06-454a-9a39-04f1fd3896d8" (UID: "51252269-ba06-454a-9a39-04f1fd3896d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:01:13.578525 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.578493 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.578525 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.578521 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.578821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.578536 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51252269-ba06-454a-9a39-04f1fd3896d8-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.578821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.578550 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.578821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.578562 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.578821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.578571 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.578821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.578579 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ee59769-d44a-4690-b5b0-aeecb07beff2-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.578821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.578588 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.578821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.578645 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zss25\" (UniqueName: \"kubernetes.io/projected/51252269-ba06-454a-9a39-04f1fd3896d8-kube-api-access-zss25\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.578821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.578654 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ee59769-d44a-4690-b5b0-aeecb07beff2-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.578821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.578663 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51252269-ba06-454a-9a39-04f1fd3896d8-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.578821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.578671 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgskr\" (UniqueName: \"kubernetes.io/projected/7ee59769-d44a-4690-b5b0-aeecb07beff2-kube-api-access-xgskr\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.578821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.578680 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51252269-ba06-454a-9a39-04f1fd3896d8-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:13.794938 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.794907 5532 generic.go:356] "Generic (PLEG): container finished" podID="51252269-ba06-454a-9a39-04f1fd3896d8" containerID="56f71a9f143c23b060c9524ec7ce3b37e9fa73f9d03d5a0dd093980feb9025db" exitCode=0 Oct 31 14:01:13.795137 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.794955 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" Oct 31 14:01:13.795137 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.794987 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" event={"ID":"51252269-ba06-454a-9a39-04f1fd3896d8","Type":"ContainerDied","Data":"56f71a9f143c23b060c9524ec7ce3b37e9fa73f9d03d5a0dd093980feb9025db"} Oct 31 14:01:13.795137 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.795041 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957" event={"ID":"51252269-ba06-454a-9a39-04f1fd3896d8","Type":"ContainerDied","Data":"0670a723837d86c79d09ba83da2854efe51ef776a2bb26b46fd5a4bc212270d6"} Oct 31 14:01:13.795137 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.795067 5532 scope.go:122] "RemoveContainer" containerID="56f71a9f143c23b060c9524ec7ce3b37e9fa73f9d03d5a0dd093980feb9025db" Oct 31 14:01:13.802630 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.796924 5532 generic.go:356] "Generic (PLEG): container finished" podID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerID="4c31911ad78912113e42a44975329378a63de2df866fa2a104b694b394a00b60" exitCode=0 Oct 31 14:01:13.802630 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.796970 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" Oct 31 14:01:13.802630 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.797012 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" event={"ID":"7ee59769-d44a-4690-b5b0-aeecb07beff2","Type":"ContainerDied","Data":"4c31911ad78912113e42a44975329378a63de2df866fa2a104b694b394a00b60"} Oct 31 14:01:13.802630 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.797066 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz" event={"ID":"7ee59769-d44a-4690-b5b0-aeecb07beff2","Type":"ContainerDied","Data":"5cfe696e7e671598042148c9aa7d5fe4f7fc9dd02a986ed49327b6dd9a7c4e36"} Oct 31 14:01:13.802630 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.798891 5532 generic.go:356] "Generic (PLEG): container finished" podID="5738e7ce-0f1c-416f-b3c5-50719400f057" containerID="1cfe1acb4e20cb8eaf181878d75b86f9047143d0e81166e1de3741458626b5f4" exitCode=0 Oct 31 14:01:13.802630 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.798927 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" Oct 31 14:01:13.802630 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.799003 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" event={"ID":"5738e7ce-0f1c-416f-b3c5-50719400f057","Type":"ContainerDied","Data":"1cfe1acb4e20cb8eaf181878d75b86f9047143d0e81166e1de3741458626b5f4"} Oct 31 14:01:13.802630 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.799052 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq" event={"ID":"5738e7ce-0f1c-416f-b3c5-50719400f057","Type":"ContainerDied","Data":"1964fa82d147667a8a1a69e5c27d0feb0b2ef270f13a6c702652f1808808fecc"} Oct 31 14:01:13.817724 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.817696 5532 scope.go:122] "RemoveContainer" containerID="56f71a9f143c23b060c9524ec7ce3b37e9fa73f9d03d5a0dd093980feb9025db" Oct 31 14:01:13.818210 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:13.818176 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56f71a9f143c23b060c9524ec7ce3b37e9fa73f9d03d5a0dd093980feb9025db\": container with ID starting with 56f71a9f143c23b060c9524ec7ce3b37e9fa73f9d03d5a0dd093980feb9025db not found: ID does not exist" containerID="56f71a9f143c23b060c9524ec7ce3b37e9fa73f9d03d5a0dd093980feb9025db" Oct 31 14:01:13.818331 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.818221 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56f71a9f143c23b060c9524ec7ce3b37e9fa73f9d03d5a0dd093980feb9025db"} err="failed to get container status \"56f71a9f143c23b060c9524ec7ce3b37e9fa73f9d03d5a0dd093980feb9025db\": rpc error: code = NotFound desc = could not find container \"56f71a9f143c23b060c9524ec7ce3b37e9fa73f9d03d5a0dd093980feb9025db\": container with ID starting with 56f71a9f143c23b060c9524ec7ce3b37e9fa73f9d03d5a0dd093980feb9025db not found: ID does not exist" Oct 31 14:01:13.818331 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.818247 5532 scope.go:122] "RemoveContainer" containerID="4c31911ad78912113e42a44975329378a63de2df866fa2a104b694b394a00b60" Oct 31 14:01:13.831719 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.831700 5532 scope.go:122] "RemoveContainer" containerID="ba38cb398433bb8c326aaf9d913b4c09baac5b5cef2b5f9c06ff230c752c872e" Oct 31 14:01:13.841703 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.841675 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq"] Oct 31 14:01:13.842089 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.842069 5532 scope.go:122] "RemoveContainer" containerID="4c31911ad78912113e42a44975329378a63de2df866fa2a104b694b394a00b60" Oct 31 14:01:13.842377 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:13.842355 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c31911ad78912113e42a44975329378a63de2df866fa2a104b694b394a00b60\": container with ID starting with 4c31911ad78912113e42a44975329378a63de2df866fa2a104b694b394a00b60 not found: ID does not exist" containerID="4c31911ad78912113e42a44975329378a63de2df866fa2a104b694b394a00b60" Oct 31 14:01:13.842442 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.842385 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c31911ad78912113e42a44975329378a63de2df866fa2a104b694b394a00b60"} err="failed to get container status \"4c31911ad78912113e42a44975329378a63de2df866fa2a104b694b394a00b60\": rpc error: code = NotFound desc = could not find container \"4c31911ad78912113e42a44975329378a63de2df866fa2a104b694b394a00b60\": container with ID starting with 4c31911ad78912113e42a44975329378a63de2df866fa2a104b694b394a00b60 not found: ID does not exist" Oct 31 14:01:13.842442 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.842403 5532 scope.go:122] "RemoveContainer" containerID="ba38cb398433bb8c326aaf9d913b4c09baac5b5cef2b5f9c06ff230c752c872e" Oct 31 14:01:13.842669 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:13.842648 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba38cb398433bb8c326aaf9d913b4c09baac5b5cef2b5f9c06ff230c752c872e\": container with ID starting with ba38cb398433bb8c326aaf9d913b4c09baac5b5cef2b5f9c06ff230c752c872e not found: ID does not exist" containerID="ba38cb398433bb8c326aaf9d913b4c09baac5b5cef2b5f9c06ff230c752c872e" Oct 31 14:01:13.842755 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.842683 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba38cb398433bb8c326aaf9d913b4c09baac5b5cef2b5f9c06ff230c752c872e"} err="failed to get container status \"ba38cb398433bb8c326aaf9d913b4c09baac5b5cef2b5f9c06ff230c752c872e\": rpc error: code = NotFound desc = could not find container \"ba38cb398433bb8c326aaf9d913b4c09baac5b5cef2b5f9c06ff230c752c872e\": container with ID starting with ba38cb398433bb8c326aaf9d913b4c09baac5b5cef2b5f9c06ff230c752c872e not found: ID does not exist" Oct 31 14:01:13.842755 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.842704 5532 scope.go:122] "RemoveContainer" containerID="1cfe1acb4e20cb8eaf181878d75b86f9047143d0e81166e1de3741458626b5f4" Oct 31 14:01:13.850143 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.850112 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-fbcgq"] Oct 31 14:01:13.853572 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.853557 5532 scope.go:122] "RemoveContainer" containerID="1cfe1acb4e20cb8eaf181878d75b86f9047143d0e81166e1de3741458626b5f4" Oct 31 14:01:13.853867 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:13.853837 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cfe1acb4e20cb8eaf181878d75b86f9047143d0e81166e1de3741458626b5f4\": container with ID starting with 1cfe1acb4e20cb8eaf181878d75b86f9047143d0e81166e1de3741458626b5f4 not found: ID does not exist" containerID="1cfe1acb4e20cb8eaf181878d75b86f9047143d0e81166e1de3741458626b5f4" Oct 31 14:01:13.853970 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.853872 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cfe1acb4e20cb8eaf181878d75b86f9047143d0e81166e1de3741458626b5f4"} err="failed to get container status \"1cfe1acb4e20cb8eaf181878d75b86f9047143d0e81166e1de3741458626b5f4\": rpc error: code = NotFound desc = could not find container \"1cfe1acb4e20cb8eaf181878d75b86f9047143d0e81166e1de3741458626b5f4\": container with ID starting with 1cfe1acb4e20cb8eaf181878d75b86f9047143d0e81166e1de3741458626b5f4 not found: ID does not exist" Oct 31 14:01:13.865018 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.864988 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957"] Oct 31 14:01:13.870403 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.870376 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-rt957"] Oct 31 14:01:13.881243 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.881219 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz"] Oct 31 14:01:13.887399 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:13.887375 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-a81979-openshift-default-7c8bcfc9bf-tb8mz"] Oct 31 14:01:14.003924 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:14.003891 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51252269-ba06-454a-9a39-04f1fd3896d8" path="/var/lib/kubelet/pods/51252269-ba06-454a-9a39-04f1fd3896d8/volumes" Oct 31 14:01:14.004408 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:14.004394 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5738e7ce-0f1c-416f-b3c5-50719400f057" path="/var/lib/kubelet/pods/5738e7ce-0f1c-416f-b3c5-50719400f057/volumes" Oct 31 14:01:14.004865 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:14.004852 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" path="/var/lib/kubelet/pods/7ee59769-d44a-4690-b5b0-aeecb07beff2/volumes" Oct 31 14:01:15.232844 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:15.232793 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" podUID="b4497eeb-2f9a-4131-b3e4-3fcef824c637" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:01:17.770620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:17.769544 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc"] Oct 31 14:01:17.770620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:17.769871 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" podUID="a5959be5-1f14-4ea5-a4b0-8d6b141052f2" containerName="payload-processing" containerID="cri-o://d78689cfce4c0a281d5df70cbffa02a6a36e6e18e57b9bd7839cbe13cc20c5b2" gracePeriod=30 Oct 31 14:01:18.144958 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.144921 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" Oct 31 14:01:18.281139 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.279275 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-plugins-config-volume\") pod \"a5959be5-1f14-4ea5-a4b0-8d6b141052f2\" (UID: \"a5959be5-1f14-4ea5-a4b0-8d6b141052f2\") " Oct 31 14:01:18.282370 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.282341 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-kube-api-access-lqskh\" (UniqueName: \"kubernetes.io/projected/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-kube-api-access-lqskh\") pod \"a5959be5-1f14-4ea5-a4b0-8d6b141052f2\" (UID: \"a5959be5-1f14-4ea5-a4b0-8d6b141052f2\") " Oct 31 14:01:18.286086 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.281116 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-plugins-config-volume" pod "a5959be5-1f14-4ea5-a4b0-8d6b141052f2" (UID: "a5959be5-1f14-4ea5-a4b0-8d6b141052f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:18.300075 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.300007 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-kube-api-access-lqskh" pod "a5959be5-1f14-4ea5-a4b0-8d6b141052f2" (UID: "a5959be5-1f14-4ea5-a4b0-8d6b141052f2"). InnerVolumeSpecName "kube-api-access-lqskh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:18.388043 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.387959 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqskh\" (UniqueName: \"kubernetes.io/projected/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-kube-api-access-lqskh\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:18.388043 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.387989 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5959be5-1f14-4ea5-a4b0-8d6b141052f2-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:18.461833 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.461802 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk"] Oct 31 14:01:18.874209 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.874173 5532 generic.go:356] "Generic (PLEG): container finished" podID="a5959be5-1f14-4ea5-a4b0-8d6b141052f2" containerID="d78689cfce4c0a281d5df70cbffa02a6a36e6e18e57b9bd7839cbe13cc20c5b2" exitCode=0 Oct 31 14:01:18.874661 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.874229 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" Oct 31 14:01:18.874661 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.874269 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" event={"ID":"a5959be5-1f14-4ea5-a4b0-8d6b141052f2","Type":"ContainerDied","Data":"d78689cfce4c0a281d5df70cbffa02a6a36e6e18e57b9bd7839cbe13cc20c5b2"} Oct 31 14:01:18.874661 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.874330 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc" event={"ID":"a5959be5-1f14-4ea5-a4b0-8d6b141052f2","Type":"ContainerDied","Data":"6ac2669b761080778858060f8a1c4c77e321eba21fb121c666d45efc3c2e9273"} Oct 31 14:01:18.874661 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.874353 5532 scope.go:122] "RemoveContainer" containerID="d78689cfce4c0a281d5df70cbffa02a6a36e6e18e57b9bd7839cbe13cc20c5b2" Oct 31 14:01:18.885543 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.885530 5532 scope.go:122] "RemoveContainer" containerID="d78689cfce4c0a281d5df70cbffa02a6a36e6e18e57b9bd7839cbe13cc20c5b2" Oct 31 14:01:18.885826 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:18.885806 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d78689cfce4c0a281d5df70cbffa02a6a36e6e18e57b9bd7839cbe13cc20c5b2\": container with ID starting with d78689cfce4c0a281d5df70cbffa02a6a36e6e18e57b9bd7839cbe13cc20c5b2 not found: ID does not exist" containerID="d78689cfce4c0a281d5df70cbffa02a6a36e6e18e57b9bd7839cbe13cc20c5b2" Oct 31 14:01:18.885905 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.885838 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d78689cfce4c0a281d5df70cbffa02a6a36e6e18e57b9bd7839cbe13cc20c5b2"} err="failed to get container status \"d78689cfce4c0a281d5df70cbffa02a6a36e6e18e57b9bd7839cbe13cc20c5b2\": rpc error: code = NotFound desc = could not find container \"d78689cfce4c0a281d5df70cbffa02a6a36e6e18e57b9bd7839cbe13cc20c5b2\": container with ID starting with d78689cfce4c0a281d5df70cbffa02a6a36e6e18e57b9bd7839cbe13cc20c5b2 not found: ID does not exist" Oct 31 14:01:18.903454 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.903424 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc"] Oct 31 14:01:18.907247 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.907227 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f9ca72-6bf6ff6f9b-qgtgc"] Oct 31 14:01:18.929230 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:18.929197 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" podUID="124cc2c6-45cd-4bcd-9714-d4041f054bbf" containerName="main" containerID="cri-o://7c53d4593bbb0b4a0f18e24e4d7a5a5d2524a83a2ce04d88ded53d159a3eb78b" gracePeriod=45 Oct 31 14:01:19.219501 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.219477 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 14:01:19.301608 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.301567 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-home\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-home\") pod \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " Oct 31 14:01:19.301800 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.301641 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tls-certs\" (UniqueName: \"kubernetes.io/secret/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tls-certs\") pod \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " Oct 31 14:01:19.301800 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.301713 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124cc2c6-45cd-4bcd-9714-d4041f054bbf-kube-api-access-mjhmx\" (UniqueName: \"kubernetes.io/projected/124cc2c6-45cd-4bcd-9714-d4041f054bbf-kube-api-access-mjhmx\") pod \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " Oct 31 14:01:19.301800 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.301773 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-dshm\") pod \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " Oct 31 14:01:19.301991 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.301835 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-home" pod "124cc2c6-45cd-4bcd-9714-d4041f054bbf" (UID: "124cc2c6-45cd-4bcd-9714-d4041f054bbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:19.301991 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.301868 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tmp-dir\") pod \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " Oct 31 14:01:19.301991 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.301922 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-model-cache\") pod \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\" (UID: \"124cc2c6-45cd-4bcd-9714-d4041f054bbf\") " Oct 31 14:01:19.302179 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.302142 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tmp-dir" pod "124cc2c6-45cd-4bcd-9714-d4041f054bbf" (UID: "124cc2c6-45cd-4bcd-9714-d4041f054bbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:19.302247 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.302191 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-model-cache" pod "124cc2c6-45cd-4bcd-9714-d4041f054bbf" (UID: "124cc2c6-45cd-4bcd-9714-d4041f054bbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:19.302440 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.302425 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:19.302520 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.302442 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:19.302520 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.302460 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:19.304002 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.303975 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-dshm" pod "124cc2c6-45cd-4bcd-9714-d4041f054bbf" (UID: "124cc2c6-45cd-4bcd-9714-d4041f054bbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:19.304100 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.304044 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124cc2c6-45cd-4bcd-9714-d4041f054bbf-kube-api-access-mjhmx" pod "124cc2c6-45cd-4bcd-9714-d4041f054bbf" (UID: "124cc2c6-45cd-4bcd-9714-d4041f054bbf"). InnerVolumeSpecName "kube-api-access-mjhmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:19.304100 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.304072 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tls-certs" pod "124cc2c6-45cd-4bcd-9714-d4041f054bbf" (UID: "124cc2c6-45cd-4bcd-9714-d4041f054bbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:01:19.404364 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.404329 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124cc2c6-45cd-4bcd-9714-d4041f054bbf-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:19.404364 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.404362 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjhmx\" (UniqueName: \"kubernetes.io/projected/124cc2c6-45cd-4bcd-9714-d4041f054bbf-kube-api-access-mjhmx\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:19.404551 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.404374 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124cc2c6-45cd-4bcd-9714-d4041f054bbf-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:19.884750 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.884715 5532 generic.go:356] "Generic (PLEG): container finished" podID="124cc2c6-45cd-4bcd-9714-d4041f054bbf" containerID="7c53d4593bbb0b4a0f18e24e4d7a5a5d2524a83a2ce04d88ded53d159a3eb78b" exitCode=0 Oct 31 14:01:19.885175 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.884763 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" Oct 31 14:01:19.885175 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.884790 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" event={"ID":"124cc2c6-45cd-4bcd-9714-d4041f054bbf","Type":"ContainerDied","Data":"7c53d4593bbb0b4a0f18e24e4d7a5a5d2524a83a2ce04d88ded53d159a3eb78b"} Oct 31 14:01:19.885175 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.884828 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp" event={"ID":"124cc2c6-45cd-4bcd-9714-d4041f054bbf","Type":"ContainerDied","Data":"ec31a9c57b216d5f7ef4f14c1b8d810f76e349a2770938e98bd099cb8792d210"} Oct 31 14:01:19.885175 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.884846 5532 scope.go:122] "RemoveContainer" containerID="7c53d4593bbb0b4a0f18e24e4d7a5a5d2524a83a2ce04d88ded53d159a3eb78b" Oct 31 14:01:19.895993 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.895976 5532 scope.go:122] "RemoveContainer" containerID="7c53d4593bbb0b4a0f18e24e4d7a5a5d2524a83a2ce04d88ded53d159a3eb78b" Oct 31 14:01:19.896240 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:19.896221 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c53d4593bbb0b4a0f18e24e4d7a5a5d2524a83a2ce04d88ded53d159a3eb78b\": container with ID starting with 7c53d4593bbb0b4a0f18e24e4d7a5a5d2524a83a2ce04d88ded53d159a3eb78b not found: ID does not exist" containerID="7c53d4593bbb0b4a0f18e24e4d7a5a5d2524a83a2ce04d88ded53d159a3eb78b" Oct 31 14:01:19.896287 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.896250 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c53d4593bbb0b4a0f18e24e4d7a5a5d2524a83a2ce04d88ded53d159a3eb78b"} err="failed to get container status \"7c53d4593bbb0b4a0f18e24e4d7a5a5d2524a83a2ce04d88ded53d159a3eb78b\": rpc error: code = NotFound desc = could not find container \"7c53d4593bbb0b4a0f18e24e4d7a5a5d2524a83a2ce04d88ded53d159a3eb78b\": container with ID starting with 7c53d4593bbb0b4a0f18e24e4d7a5a5d2524a83a2ce04d88ded53d159a3eb78b not found: ID does not exist" Oct 31 14:01:19.914839 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.914810 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp"] Oct 31 14:01:19.920226 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:19.920204 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-974711/rate-test-model-974711-kserve-6b6476cfdf-6bnxp"] Oct 31 14:01:20.003915 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:20.003881 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124cc2c6-45cd-4bcd-9714-d4041f054bbf" path="/var/lib/kubelet/pods/124cc2c6-45cd-4bcd-9714-d4041f054bbf/volumes" Oct 31 14:01:20.004371 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:20.004355 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5959be5-1f14-4ea5-a4b0-8d6b141052f2" path="/var/lib/kubelet/pods/a5959be5-1f14-4ea5-a4b0-8d6b141052f2/volumes" Oct 31 14:01:22.144728 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:22.144695 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4"] Oct 31 14:01:22.150053 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:22.150018 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-599291-glzz4"] Oct 31 14:01:23.462433 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:23.462385 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" podUID="8ad79fea-9683-4d69-a99b-77e69db4bb68" containerName="payload-pre-processing" containerID="cri-o://21c4eb83f644f0d81a17c405c5d3144269e2945413cf830b2ebeb9460f367cd8" gracePeriod=25 Oct 31 14:01:24.005929 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:24.005896 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee7986d3-2b66-4be6-9bb6-16093b0d0dde" path="/var/lib/kubelet/pods/ee7986d3-2b66-4be6-9bb6-16093b0d0dde/volumes" Oct 31 14:01:25.232886 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:25.232845 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" podUID="b4497eeb-2f9a-4131-b3e4-3fcef824c637" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:01:26.626237 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.626195 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z"] Oct 31 14:01:26.627704 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627675 5532 cpu_manager.go:414] "containerMap: removing container" podUID="61795f0c-b2b7-46a8-95b8-5f132f863cf7" containerName="main" Oct 31 14:01:26.627704 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627703 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61795f0c-b2b7-46a8-95b8-5f132f863cf7" containerName="main" Oct 31 14:01:26.627931 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627731 5532 cpu_manager.go:414] "containerMap: removing container" podUID="124cc2c6-45cd-4bcd-9714-d4041f054bbf" containerName="main" Oct 31 14:01:26.627931 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627739 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124cc2c6-45cd-4bcd-9714-d4041f054bbf" containerName="main" Oct 31 14:01:26.627931 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627761 5532 cpu_manager.go:414] "containerMap: removing container" podUID="5738e7ce-0f1c-416f-b3c5-50719400f057" containerName="istio-proxy" Oct 31 14:01:26.627931 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627769 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5738e7ce-0f1c-416f-b3c5-50719400f057" containerName="istio-proxy" Oct 31 14:01:26.627931 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627831 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a5959be5-1f14-4ea5-a4b0-8d6b141052f2" containerName="payload-processing" Oct 31 14:01:26.627931 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627840 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5959be5-1f14-4ea5-a4b0-8d6b141052f2" containerName="payload-processing" Oct 31 14:01:26.627931 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627858 5532 cpu_manager.go:414] "containerMap: removing container" podUID="79127725-b5df-4400-b000-37bbabd42727" containerName="revoke-keys" Oct 31 14:01:26.627931 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627866 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79127725-b5df-4400-b000-37bbabd42727" containerName="revoke-keys" Oct 31 14:01:26.627931 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627899 5532 cpu_manager.go:414] "containerMap: removing container" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" Oct 31 14:01:26.627931 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627907 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" Oct 31 14:01:26.628429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627965 5532 cpu_manager.go:414] "containerMap: removing container" podUID="51252269-ba06-454a-9a39-04f1fd3896d8" containerName="istio-proxy" Oct 31 14:01:26.628429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.627974 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51252269-ba06-454a-9a39-04f1fd3896d8" containerName="istio-proxy" Oct 31 14:01:26.628429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.628317 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="124cc2c6-45cd-4bcd-9714-d4041f054bbf" containerName="main" Oct 31 14:01:26.628429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.628353 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="5738e7ce-0f1c-416f-b3c5-50719400f057" containerName="istio-proxy" Oct 31 14:01:26.628429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.628378 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5959be5-1f14-4ea5-a4b0-8d6b141052f2" containerName="payload-processing" Oct 31 14:01:26.628429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.628389 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="79127725-b5df-4400-b000-37bbabd42727" containerName="revoke-keys" Oct 31 14:01:26.628429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.628400 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" Oct 31 14:01:26.628429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.628421 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="51252269-ba06-454a-9a39-04f1fd3896d8" containerName="istio-proxy" Oct 31 14:01:26.628845 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.628439 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" Oct 31 14:01:26.628845 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.628452 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="61795f0c-b2b7-46a8-95b8-5f132f863cf7" containerName="main" Oct 31 14:01:26.636649 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.636584 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" Oct 31 14:01:26.638588 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.638562 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z"] Oct 31 14:01:26.641874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.641797 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:01:26.648832 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.648801 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:01:26.725073 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.725045 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z\" (UID: \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" Oct 31 14:01:26.725219 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.725181 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z\" (UID: \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" Oct 31 14:01:26.725286 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.725268 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwdbt\" (UniqueName: \"kubernetes.io/projected/e3a4ae30-4d47-420b-bde7-c7174eabaedf-kube-api-access-dwdbt\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z\" (UID: \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" Oct 31 14:01:26.826687 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.826652 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z\" (UID: \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" Oct 31 14:01:26.826871 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.826771 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z\" (UID: \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" Oct 31 14:01:26.826871 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.826826 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwdbt\" (UniqueName: \"kubernetes.io/projected/e3a4ae30-4d47-420b-bde7-c7174eabaedf-kube-api-access-dwdbt\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z\" (UID: \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" Oct 31 14:01:26.827432 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.827403 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z\" (UID: \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" Oct 31 14:01:26.829285 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.829260 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z\" (UID: \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" Oct 31 14:01:26.840100 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.840071 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwdbt\" (UniqueName: \"kubernetes.io/projected/e3a4ae30-4d47-420b-bde7-c7174eabaedf-kube-api-access-dwdbt\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z\" (UID: \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" Oct 31 14:01:26.961551 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:26.961518 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" Oct 31 14:01:27.272721 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:27.272691 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z"] Oct 31 14:01:27.273826 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:01:27.273799 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3a4ae30_4d47_420b_bde7_c7174eabaedf.slice/crio-44e61dc9999f836cbb6965e72f3870ac8e59fd59a4ec599f809ce90f050c577a WatchSource:0}: Error finding container 44e61dc9999f836cbb6965e72f3870ac8e59fd59a4ec599f809ce90f050c577a: Status 404 returned error can't find the container with id 44e61dc9999f836cbb6965e72f3870ac8e59fd59a4ec599f809ce90f050c577a Oct 31 14:01:27.954235 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:27.954191 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" event={"ID":"e3a4ae30-4d47-420b-bde7-c7174eabaedf","Type":"ContainerStarted","Data":"7d49a3590be14a5ae0843886ce1daf5f8213e37a98bccf8c6143ea7470c0bf26"} Oct 31 14:01:27.954858 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:27.954829 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" event={"ID":"e3a4ae30-4d47-420b-bde7-c7174eabaedf","Type":"ContainerStarted","Data":"44e61dc9999f836cbb6965e72f3870ac8e59fd59a4ec599f809ce90f050c577a"} Oct 31 14:01:27.973307 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:27.973271 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" podUID="8ad79fea-9683-4d69-a99b-77e69db4bb68" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:01:27.986640 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:27.986572 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" podStartSLOduration=1.986558842 podStartE2EDuration="1.986558842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:27.982552189 +0000 UTC m=+3456.128296016" watchObservedRunningTime="2026-10-31 14:01:27.986558842 +0000 UTC m=+3456.132302669" Oct 31 14:01:28.965190 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:28.965087 5532 generic.go:356] "Generic (PLEG): container finished" podID="e3a4ae30-4d47-420b-bde7-c7174eabaedf" containerID="7d49a3590be14a5ae0843886ce1daf5f8213e37a98bccf8c6143ea7470c0bf26" exitCode=0 Oct 31 14:01:28.966299 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:28.966250 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" event={"ID":"e3a4ae30-4d47-420b-bde7-c7174eabaedf","Type":"ContainerDied","Data":"7d49a3590be14a5ae0843886ce1daf5f8213e37a98bccf8c6143ea7470c0bf26"} Oct 31 14:01:30.240838 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.240776 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" Oct 31 14:01:30.373230 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.373192 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-xks-ca\") pod \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\" (UID: \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\") " Oct 31 14:01:30.373456 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.373288 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-openshift-ca\") pod \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\" (UID: \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\") " Oct 31 14:01:30.373456 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.373352 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3a4ae30-4d47-420b-bde7-c7174eabaedf-kube-api-access-dwdbt\" (UniqueName: \"kubernetes.io/projected/e3a4ae30-4d47-420b-bde7-c7174eabaedf-kube-api-access-dwdbt\") pod \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\" (UID: \"e3a4ae30-4d47-420b-bde7-c7174eabaedf\") " Oct 31 14:01:30.373687 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.373659 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-openshift-ca" pod "e3a4ae30-4d47-420b-bde7-c7174eabaedf" (UID: "e3a4ae30-4d47-420b-bde7-c7174eabaedf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:30.374090 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.374063 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:30.375811 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.375758 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3a4ae30-4d47-420b-bde7-c7174eabaedf-kube-api-access-dwdbt" pod "e3a4ae30-4d47-420b-bde7-c7174eabaedf" (UID: "e3a4ae30-4d47-420b-bde7-c7174eabaedf"). InnerVolumeSpecName "kube-api-access-dwdbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:30.377992 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.377964 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-xks-ca" pod "e3a4ae30-4d47-420b-bde7-c7174eabaedf" (UID: "e3a4ae30-4d47-420b-bde7-c7174eabaedf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:01:30.447633 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.447571 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt"] Oct 31 14:01:30.447866 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.447840 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" podUID="279335a6-fc12-417b-824a-9ceafe2d3431" containerName="istio-proxy" containerID="cri-o://b88d1347fe3624c93559a0aa42e61a11093681c0d9205b5349e3c754be09e11c" gracePeriod=30 Oct 31 14:01:30.475583 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.475548 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3a4ae30-4d47-420b-bde7-c7174eabaedf-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:30.475583 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.475583 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwdbt\" (UniqueName: \"kubernetes.io/projected/e3a4ae30-4d47-420b-bde7-c7174eabaedf-kube-api-access-dwdbt\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:30.986380 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.986237 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" Oct 31 14:01:30.986558 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.986441 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z" event={"ID":"e3a4ae30-4d47-420b-bde7-c7174eabaedf","Type":"ContainerDied","Data":"44e61dc9999f836cbb6965e72f3870ac8e59fd59a4ec599f809ce90f050c577a"} Oct 31 14:01:30.986558 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.986500 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44e61dc9999f836cbb6965e72f3870ac8e59fd59a4ec599f809ce90f050c577a" Oct 31 14:01:30.992868 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.992839 5532 generic.go:356] "Generic (PLEG): container finished" podID="b4497eeb-2f9a-4131-b3e4-3fcef824c637" containerID="09e29943cee80b121099b7438491b95e63ede76290297e477a5cf0d99d5882f0" exitCode=0 Oct 31 14:01:30.992998 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:30.992980 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" event={"ID":"b4497eeb-2f9a-4131-b3e4-3fcef824c637","Type":"ContainerDied","Data":"09e29943cee80b121099b7438491b95e63ede76290297e477a5cf0d99d5882f0"} Oct 31 14:01:31.064838 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.064810 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" Oct 31 14:01:31.087910 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.087833 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx"] Oct 31 14:01:31.088754 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.088737 5532 cpu_manager.go:414] "containerMap: removing container" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" Oct 31 14:01:31.088823 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.088755 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ee59769-d44a-4690-b5b0-aeecb07beff2" containerName="istio-proxy" Oct 31 14:01:31.088823 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.088783 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b4497eeb-2f9a-4131-b3e4-3fcef824c637" containerName="payload-pre-processing" Oct 31 14:01:31.088823 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.088791 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4497eeb-2f9a-4131-b3e4-3fcef824c637" containerName="payload-pre-processing" Oct 31 14:01:31.088823 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.088817 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e3a4ae30-4d47-420b-bde7-c7174eabaedf" containerName="revoke-keys" Oct 31 14:01:31.088974 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.088824 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3a4ae30-4d47-420b-bde7-c7174eabaedf" containerName="revoke-keys" Oct 31 14:01:31.088974 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.088954 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4497eeb-2f9a-4131-b3e4-3fcef824c637" containerName="payload-pre-processing" Oct 31 14:01:31.088974 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.088969 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3a4ae30-4d47-420b-bde7-c7174eabaedf" containerName="revoke-keys" Oct 31 14:01:31.092845 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.092825 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" Oct 31 14:01:31.094856 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.094833 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4497eeb-2f9a-4131-b3e4-3fcef824c637-kube-api-access-f4l8h\" (UniqueName: \"kubernetes.io/projected/b4497eeb-2f9a-4131-b3e4-3fcef824c637-kube-api-access-f4l8h\") pod \"b4497eeb-2f9a-4131-b3e4-3fcef824c637\" (UID: \"b4497eeb-2f9a-4131-b3e4-3fcef824c637\") " Oct 31 14:01:31.095007 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.094989 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4497eeb-2f9a-4131-b3e4-3fcef824c637-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4497eeb-2f9a-4131-b3e4-3fcef824c637-plugins-config-volume\") pod \"b4497eeb-2f9a-4131-b3e4-3fcef824c637\" (UID: \"b4497eeb-2f9a-4131-b3e4-3fcef824c637\") " Oct 31 14:01:31.095378 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.095351 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4497eeb-2f9a-4131-b3e4-3fcef824c637-plugins-config-volume" pod "b4497eeb-2f9a-4131-b3e4-3fcef824c637" (UID: "b4497eeb-2f9a-4131-b3e4-3fcef824c637"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:31.096433 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.096412 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4497eeb-2f9a-4131-b3e4-3fcef824c637-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:31.097472 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.097443 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4497eeb-2f9a-4131-b3e4-3fcef824c637-kube-api-access-f4l8h" pod "b4497eeb-2f9a-4131-b3e4-3fcef824c637" (UID: "b4497eeb-2f9a-4131-b3e4-3fcef824c637"). InnerVolumeSpecName "kube-api-access-f4l8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:31.119304 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.119280 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx"] Oct 31 14:01:31.157757 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.157731 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:01:31.157919 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.157792 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:01:31.198071 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.198043 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkg6\" (UniqueName: \"kubernetes.io/projected/df332b53-ff14-4e58-b911-dd1659c28d87-kube-api-access-hnkg6\") pod \"maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx\" (UID: \"df332b53-ff14-4e58-b911-dd1659c28d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" Oct 31 14:01:31.198275 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.198260 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx\" (UID: \"df332b53-ff14-4e58-b911-dd1659c28d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" Oct 31 14:01:31.198333 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.198305 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx\" (UID: \"df332b53-ff14-4e58-b911-dd1659c28d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" Oct 31 14:01:31.198470 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.198458 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f4l8h\" (UniqueName: \"kubernetes.io/projected/b4497eeb-2f9a-4131-b3e4-3fcef824c637-kube-api-access-f4l8h\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:31.301238 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.300482 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx\" (UID: \"df332b53-ff14-4e58-b911-dd1659c28d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" Oct 31 14:01:31.301238 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.300663 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkg6\" (UniqueName: \"kubernetes.io/projected/df332b53-ff14-4e58-b911-dd1659c28d87-kube-api-access-hnkg6\") pod \"maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx\" (UID: \"df332b53-ff14-4e58-b911-dd1659c28d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" Oct 31 14:01:31.301238 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.300899 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx\" (UID: \"df332b53-ff14-4e58-b911-dd1659c28d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" Oct 31 14:01:31.310383 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.308485 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx\" (UID: \"df332b53-ff14-4e58-b911-dd1659c28d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" Oct 31 14:01:31.313254 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.313219 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx\" (UID: \"df332b53-ff14-4e58-b911-dd1659c28d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" Oct 31 14:01:31.322168 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.322141 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkg6\" (UniqueName: \"kubernetes.io/projected/df332b53-ff14-4e58-b911-dd1659c28d87-kube-api-access-hnkg6\") pod \"maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx\" (UID: \"df332b53-ff14-4e58-b911-dd1659c28d87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" Oct 31 14:01:31.409369 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.409336 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" Oct 31 14:01:31.697925 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:31.697897 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx"] Oct 31 14:01:31.698483 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:01:31.698463 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf332b53_ff14_4e58_b911_dd1659c28d87.slice/crio-333ced78b7c4ee10eb8344973a5a831554d8335dff309e49dd4dcc8df63efbde WatchSource:0}: Error finding container 333ced78b7c4ee10eb8344973a5a831554d8335dff309e49dd4dcc8df63efbde: Status 404 returned error can't find the container with id 333ced78b7c4ee10eb8344973a5a831554d8335dff309e49dd4dcc8df63efbde Oct 31 14:01:31.761720 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:31.761678 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:01:32.004822 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:32.004789 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" event={"ID":"b4497eeb-2f9a-4131-b3e4-3fcef824c637","Type":"ContainerDied","Data":"9397105b4b37c441598417ba6caa688379ffe6bb1c91fe407028bb12ae0cb79f"} Oct 31 14:01:32.005005 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:32.004852 5532 scope.go:122] "RemoveContainer" containerID="09e29943cee80b121099b7438491b95e63ede76290297e477a5cf0d99d5882f0" Oct 31 14:01:32.005005 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:32.004797 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh" Oct 31 14:01:32.006724 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:32.006694 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" event={"ID":"df332b53-ff14-4e58-b911-dd1659c28d87","Type":"ContainerStarted","Data":"95b832975fb32221bbb5f8bac0f874376d2307bf7540e8fd3da5a9452fcef12b"} Oct 31 14:01:32.006854 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:32.006755 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" event={"ID":"df332b53-ff14-4e58-b911-dd1659c28d87","Type":"ContainerStarted","Data":"333ced78b7c4ee10eb8344973a5a831554d8335dff309e49dd4dcc8df63efbde"} Oct 31 14:01:32.032901 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:32.032855 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" podStartSLOduration=1.032843256 podStartE2EDuration="1.032843256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:32.028619428 +0000 UTC m=+3460.174363258" watchObservedRunningTime="2026-10-31 14:01:32.032843256 +0000 UTC m=+3460.178587084" Oct 31 14:01:32.046559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:32.046535 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh"] Oct 31 14:01:32.051052 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:32.051024 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-599291-7b7c7fccf9-bqvfh"] Oct 31 14:01:32.484967 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:32.484930 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:01:32.485334 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:32.485001 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" podUID="279335a6-fc12-417b-824a-9ceafe2d3431" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:01:33.019205 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:33.019169 5532 generic.go:356] "Generic (PLEG): container finished" podID="df332b53-ff14-4e58-b911-dd1659c28d87" containerID="95b832975fb32221bbb5f8bac0f874376d2307bf7540e8fd3da5a9452fcef12b" exitCode=0 Oct 31 14:01:33.019414 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:33.019266 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" event={"ID":"df332b53-ff14-4e58-b911-dd1659c28d87","Type":"ContainerDied","Data":"95b832975fb32221bbb5f8bac0f874376d2307bf7540e8fd3da5a9452fcef12b"} Oct 31 14:01:33.102477 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:33.102430 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl"] Oct 31 14:01:33.113077 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:33.113051 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-403dff-ssmsl"] Oct 31 14:01:34.006807 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:34.006772 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79127725-b5df-4400-b000-37bbabd42727" path="/var/lib/kubelet/pods/79127725-b5df-4400-b000-37bbabd42727/volumes" Oct 31 14:01:34.007523 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:34.007449 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4497eeb-2f9a-4131-b3e4-3fcef824c637" path="/var/lib/kubelet/pods/b4497eeb-2f9a-4131-b3e4-3fcef824c637/volumes" Oct 31 14:01:34.199174 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:34.199148 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" Oct 31 14:01:34.257771 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:34.257690 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-xks-ca\") pod \"df332b53-ff14-4e58-b911-dd1659c28d87\" (UID: \"df332b53-ff14-4e58-b911-dd1659c28d87\") " Oct 31 14:01:34.257771 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:34.257754 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-openshift-ca\") pod \"df332b53-ff14-4e58-b911-dd1659c28d87\" (UID: \"df332b53-ff14-4e58-b911-dd1659c28d87\") " Oct 31 14:01:34.258017 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:34.257802 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df332b53-ff14-4e58-b911-dd1659c28d87-kube-api-access-hnkg6\" (UniqueName: \"kubernetes.io/projected/df332b53-ff14-4e58-b911-dd1659c28d87-kube-api-access-hnkg6\") pod \"df332b53-ff14-4e58-b911-dd1659c28d87\" (UID: \"df332b53-ff14-4e58-b911-dd1659c28d87\") " Oct 31 14:01:34.258221 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:34.258187 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-openshift-ca" pod "df332b53-ff14-4e58-b911-dd1659c28d87" (UID: "df332b53-ff14-4e58-b911-dd1659c28d87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:34.258522 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:34.258500 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:34.260482 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:34.260444 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-xks-ca" pod "df332b53-ff14-4e58-b911-dd1659c28d87" (UID: "df332b53-ff14-4e58-b911-dd1659c28d87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:01:34.260809 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:34.260779 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df332b53-ff14-4e58-b911-dd1659c28d87-kube-api-access-hnkg6" pod "df332b53-ff14-4e58-b911-dd1659c28d87" (UID: "df332b53-ff14-4e58-b911-dd1659c28d87"). InnerVolumeSpecName "kube-api-access-hnkg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:34.361966 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:34.361921 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df332b53-ff14-4e58-b911-dd1659c28d87-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:34.361966 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:34.361964 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnkg6\" (UniqueName: \"kubernetes.io/projected/df332b53-ff14-4e58-b911-dd1659c28d87-kube-api-access-hnkg6\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:35.058621 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.057697 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" event={"ID":"df332b53-ff14-4e58-b911-dd1659c28d87","Type":"ContainerDied","Data":"333ced78b7c4ee10eb8344973a5a831554d8335dff309e49dd4dcc8df63efbde"} Oct 31 14:01:35.058621 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.057764 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="333ced78b7c4ee10eb8344973a5a831554d8335dff309e49dd4dcc8df63efbde" Oct 31 14:01:35.058621 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.057842 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx" Oct 31 14:01:35.723244 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.723179 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 14:01:35.781975 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.781943 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-data\") pod \"279335a6-fc12-417b-824a-9ceafe2d3431\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " Oct 31 14:01:35.782149 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.781997 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/279335a6-fc12-417b-824a-9ceafe2d3431-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279335a6-fc12-417b-824a-9ceafe2d3431-istio-podinfo\") pod \"279335a6-fc12-417b-824a-9ceafe2d3431\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " Oct 31 14:01:35.782149 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.782049 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-istio-token\" (UniqueName: \"kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-istio-token\") pod \"279335a6-fc12-417b-824a-9ceafe2d3431\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " Oct 31 14:01:35.782149 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.782133 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-kube-api-access-ccz4b\" (UniqueName: \"kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-kube-api-access-ccz4b\") pod \"279335a6-fc12-417b-824a-9ceafe2d3431\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " Oct 31 14:01:35.782368 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.782167 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-certs\") pod \"279335a6-fc12-417b-824a-9ceafe2d3431\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " Oct 31 14:01:35.782368 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.782194 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/279335a6-fc12-417b-824a-9ceafe2d3431-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279335a6-fc12-417b-824a-9ceafe2d3431-istiod-ca-cert\") pod \"279335a6-fc12-417b-824a-9ceafe2d3431\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " Oct 31 14:01:35.782368 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.782231 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-envoy\") pod \"279335a6-fc12-417b-824a-9ceafe2d3431\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " Oct 31 14:01:35.782368 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.782268 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-data" pod "279335a6-fc12-417b-824a-9ceafe2d3431" (UID: "279335a6-fc12-417b-824a-9ceafe2d3431"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:35.782368 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.782343 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-credential-socket\") pod \"279335a6-fc12-417b-824a-9ceafe2d3431\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " Oct 31 14:01:35.782661 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.782457 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-socket\") pod \"279335a6-fc12-417b-824a-9ceafe2d3431\" (UID: \"279335a6-fc12-417b-824a-9ceafe2d3431\") " Oct 31 14:01:35.782661 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.782502 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-certs" pod "279335a6-fc12-417b-824a-9ceafe2d3431" (UID: "279335a6-fc12-417b-824a-9ceafe2d3431"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:35.782946 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.782924 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-credential-socket" pod "279335a6-fc12-417b-824a-9ceafe2d3431" (UID: "279335a6-fc12-417b-824a-9ceafe2d3431"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:35.783079 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.783051 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-socket" pod "279335a6-fc12-417b-824a-9ceafe2d3431" (UID: "279335a6-fc12-417b-824a-9ceafe2d3431"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:35.783137 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.783097 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279335a6-fc12-417b-824a-9ceafe2d3431-istiod-ca-cert" pod "279335a6-fc12-417b-824a-9ceafe2d3431" (UID: "279335a6-fc12-417b-824a-9ceafe2d3431"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:35.783200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.783163 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:35.783200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.783187 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:35.783319 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.783204 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:35.784772 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.784731 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/279335a6-fc12-417b-824a-9ceafe2d3431-istio-podinfo" pod "279335a6-fc12-417b-824a-9ceafe2d3431" (UID: "279335a6-fc12-417b-824a-9ceafe2d3431"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:01:35.785274 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.785243 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-envoy" pod "279335a6-fc12-417b-824a-9ceafe2d3431" (UID: "279335a6-fc12-417b-824a-9ceafe2d3431"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:35.785274 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.785254 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-kube-api-access-ccz4b" pod "279335a6-fc12-417b-824a-9ceafe2d3431" (UID: "279335a6-fc12-417b-824a-9ceafe2d3431"). InnerVolumeSpecName "kube-api-access-ccz4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:35.785684 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.785660 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-istio-token" pod "279335a6-fc12-417b-824a-9ceafe2d3431" (UID: "279335a6-fc12-417b-824a-9ceafe2d3431"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:35.884342 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.884305 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:35.884342 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.884338 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/279335a6-fc12-417b-824a-9ceafe2d3431-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:35.884342 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.884347 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:35.884631 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.884356 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccz4b\" (UniqueName: \"kubernetes.io/projected/279335a6-fc12-417b-824a-9ceafe2d3431-kube-api-access-ccz4b\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:35.884631 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.884367 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/279335a6-fc12-417b-824a-9ceafe2d3431-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:35.884631 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.884375 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/279335a6-fc12-417b-824a-9ceafe2d3431-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:35.949695 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.949658 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q"] Oct 31 14:01:35.964730 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:35.964695 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd"] Oct 31 14:01:36.072775 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:36.072737 5532 generic.go:356] "Generic (PLEG): container finished" podID="279335a6-fc12-417b-824a-9ceafe2d3431" containerID="b88d1347fe3624c93559a0aa42e61a11093681c0d9205b5349e3c754be09e11c" exitCode=0 Oct 31 14:01:36.073231 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:36.072933 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" Oct 31 14:01:36.073231 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:36.073066 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" event={"ID":"279335a6-fc12-417b-824a-9ceafe2d3431","Type":"ContainerDied","Data":"b88d1347fe3624c93559a0aa42e61a11093681c0d9205b5349e3c754be09e11c"} Oct 31 14:01:36.073231 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:36.073143 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt" event={"ID":"279335a6-fc12-417b-824a-9ceafe2d3431","Type":"ContainerDied","Data":"27d8c486811e5751b856298e7c64e24075b059f9e82731bd305c999b6c4a8edb"} Oct 31 14:01:36.073231 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:36.073175 5532 scope.go:122] "RemoveContainer" containerID="b88d1347fe3624c93559a0aa42e61a11093681c0d9205b5349e3c754be09e11c" Oct 31 14:01:36.085225 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:36.085206 5532 scope.go:122] "RemoveContainer" containerID="b88d1347fe3624c93559a0aa42e61a11093681c0d9205b5349e3c754be09e11c" Oct 31 14:01:36.085574 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:36.085545 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b88d1347fe3624c93559a0aa42e61a11093681c0d9205b5349e3c754be09e11c\": container with ID starting with b88d1347fe3624c93559a0aa42e61a11093681c0d9205b5349e3c754be09e11c not found: ID does not exist" containerID="b88d1347fe3624c93559a0aa42e61a11093681c0d9205b5349e3c754be09e11c" Oct 31 14:01:36.085714 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:36.085584 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b88d1347fe3624c93559a0aa42e61a11093681c0d9205b5349e3c754be09e11c"} err="failed to get container status \"b88d1347fe3624c93559a0aa42e61a11093681c0d9205b5349e3c754be09e11c\": rpc error: code = NotFound desc = could not find container \"b88d1347fe3624c93559a0aa42e61a11093681c0d9205b5349e3c754be09e11c\": container with ID starting with b88d1347fe3624c93559a0aa42e61a11093681c0d9205b5349e3c754be09e11c not found: ID does not exist" Oct 31 14:01:36.097508 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:36.097485 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt"] Oct 31 14:01:36.102186 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:36.102167 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-599291-openshift-default-69f444c65c-zv8nt"] Oct 31 14:01:36.680015 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:36.679981 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f"] Oct 31 14:01:37.972896 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:37.972857 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" podUID="8ad79fea-9683-4d69-a99b-77e69db4bb68" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:01:38.004284 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:38.004252 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="279335a6-fc12-417b-824a-9ceafe2d3431" path="/var/lib/kubelet/pods/279335a6-fc12-417b-824a-9ceafe2d3431/volumes" Oct 31 14:01:41.131838 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:41.131807 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs"] Oct 31 14:01:41.682039 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:41.681966 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" podUID="de6b3aca-3802-48c4-9315-2492f01492a0" containerName="payload-processing" containerID="cri-o://7ba608848fc44cfe24b5895340fb2dacd05f9085ac4ab9130847d3cd6fff3511" gracePeriod=25 Oct 31 14:01:41.697192 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:41.697159 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" podUID="de6b3aca-3802-48c4-9315-2492f01492a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:01:41.972065 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:41.971989 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2"] Oct 31 14:01:41.972285 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:41.972255 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" containerID="cri-o://c33cc72858300afe60d86c3981ceba0c38a044e698a8c2d41b308acde7e8a51c" gracePeriod=30 Oct 31 14:01:42.766950 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.766916 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd"] Oct 31 14:01:42.767523 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.767508 5532 cpu_manager.go:414] "containerMap: removing container" podUID="df332b53-ff14-4e58-b911-dd1659c28d87" containerName="revoke-keys" Oct 31 14:01:42.767573 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.767524 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df332b53-ff14-4e58-b911-dd1659c28d87" containerName="revoke-keys" Oct 31 14:01:42.767573 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.767544 5532 cpu_manager.go:414] "containerMap: removing container" podUID="279335a6-fc12-417b-824a-9ceafe2d3431" containerName="istio-proxy" Oct 31 14:01:42.767573 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.767551 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="279335a6-fc12-417b-824a-9ceafe2d3431" containerName="istio-proxy" Oct 31 14:01:42.767759 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.767685 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="df332b53-ff14-4e58-b911-dd1659c28d87" containerName="revoke-keys" Oct 31 14:01:42.767759 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.767696 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="279335a6-fc12-417b-824a-9ceafe2d3431" containerName="istio-proxy" Oct 31 14:01:42.772407 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.772391 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" Oct 31 14:01:42.783552 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.783528 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd"] Oct 31 14:01:42.809634 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.809585 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxq78\" (UniqueName: \"kubernetes.io/projected/754e2b2d-0972-4bf4-bba4-a703d294e815-kube-api-access-xxq78\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd\" (UID: \"754e2b2d-0972-4bf4-bba4-a703d294e815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" Oct 31 14:01:42.809803 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.809761 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd\" (UID: \"754e2b2d-0972-4bf4-bba4-a703d294e815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" Oct 31 14:01:42.809881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.809807 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd\" (UID: \"754e2b2d-0972-4bf4-bba4-a703d294e815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" Oct 31 14:01:42.841239 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.841210 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:01:42.841517 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.841503 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:01:42.911859 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.911821 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxq78\" (UniqueName: \"kubernetes.io/projected/754e2b2d-0972-4bf4-bba4-a703d294e815-kube-api-access-xxq78\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd\" (UID: \"754e2b2d-0972-4bf4-bba4-a703d294e815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" Oct 31 14:01:42.912043 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.911921 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd\" (UID: \"754e2b2d-0972-4bf4-bba4-a703d294e815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" Oct 31 14:01:42.912043 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.911947 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd\" (UID: \"754e2b2d-0972-4bf4-bba4-a703d294e815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" Oct 31 14:01:42.912584 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.912556 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd\" (UID: \"754e2b2d-0972-4bf4-bba4-a703d294e815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" Oct 31 14:01:42.914508 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.914478 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd\" (UID: \"754e2b2d-0972-4bf4-bba4-a703d294e815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" Oct 31 14:01:42.925585 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:42.925561 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxq78\" (UniqueName: \"kubernetes.io/projected/754e2b2d-0972-4bf4-bba4-a703d294e815-kube-api-access-xxq78\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd\" (UID: \"754e2b2d-0972-4bf4-bba4-a703d294e815\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" Oct 31 14:01:43.089064 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:43.088977 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" Oct 31 14:01:43.376181 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:01:43.376143 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod754e2b2d_0972_4bf4_bba4_a703d294e815.slice/crio-36261ae9e6e1172b9edb58dc77d4736eb8b8e4d54ba8d8e7c96c299e835572b9 WatchSource:0}: Error finding container 36261ae9e6e1172b9edb58dc77d4736eb8b8e4d54ba8d8e7c96c299e835572b9: Status 404 returned error can't find the container with id 36261ae9e6e1172b9edb58dc77d4736eb8b8e4d54ba8d8e7c96c299e835572b9 Oct 31 14:01:43.376402 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:43.376378 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd"] Oct 31 14:01:43.726761 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:43.726737 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" Oct 31 14:01:43.823081 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:43.823051 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad79fea-9683-4d69-a99b-77e69db4bb68-kube-api-access-jzlc6\" (UniqueName: \"kubernetes.io/projected/8ad79fea-9683-4d69-a99b-77e69db4bb68-kube-api-access-jzlc6\") pod \"8ad79fea-9683-4d69-a99b-77e69db4bb68\" (UID: \"8ad79fea-9683-4d69-a99b-77e69db4bb68\") " Oct 31 14:01:43.823568 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:43.823203 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad79fea-9683-4d69-a99b-77e69db4bb68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad79fea-9683-4d69-a99b-77e69db4bb68-plugins-config-volume\") pod \"8ad79fea-9683-4d69-a99b-77e69db4bb68\" (UID: \"8ad79fea-9683-4d69-a99b-77e69db4bb68\") " Oct 31 14:01:43.823695 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:43.823561 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad79fea-9683-4d69-a99b-77e69db4bb68-plugins-config-volume" pod "8ad79fea-9683-4d69-a99b-77e69db4bb68" (UID: "8ad79fea-9683-4d69-a99b-77e69db4bb68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:43.823981 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:43.823950 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ad79fea-9683-4d69-a99b-77e69db4bb68-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:43.825173 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:43.825155 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad79fea-9683-4d69-a99b-77e69db4bb68-kube-api-access-jzlc6" pod "8ad79fea-9683-4d69-a99b-77e69db4bb68" (UID: "8ad79fea-9683-4d69-a99b-77e69db4bb68"). InnerVolumeSpecName "kube-api-access-jzlc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:43.869573 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:43.869535 5532 patch_prober.go:29] interesting pod/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:01:43.869919 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:43.869611 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:01:43.925932 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:43.925893 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzlc6\" (UniqueName: \"kubernetes.io/projected/8ad79fea-9683-4d69-a99b-77e69db4bb68-kube-api-access-jzlc6\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:44.215358 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:44.215272 5532 generic.go:356] "Generic (PLEG): container finished" podID="8ad79fea-9683-4d69-a99b-77e69db4bb68" containerID="21c4eb83f644f0d81a17c405c5d3144269e2945413cf830b2ebeb9460f367cd8" exitCode=0 Oct 31 14:01:44.215605 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:44.215568 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" event={"ID":"8ad79fea-9683-4d69-a99b-77e69db4bb68","Type":"ContainerDied","Data":"21c4eb83f644f0d81a17c405c5d3144269e2945413cf830b2ebeb9460f367cd8"} Oct 31 14:01:44.215725 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:44.215672 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" event={"ID":"8ad79fea-9683-4d69-a99b-77e69db4bb68","Type":"ContainerDied","Data":"8cd15dd477f51263f4472d5edf67d35b7be78713315a3047ae569ca0df8dc705"} Oct 31 14:01:44.215725 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:44.215709 5532 scope.go:122] "RemoveContainer" containerID="21c4eb83f644f0d81a17c405c5d3144269e2945413cf830b2ebeb9460f367cd8" Oct 31 14:01:44.215917 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:44.215900 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk" Oct 31 14:01:44.232657 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:44.230617 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" event={"ID":"754e2b2d-0972-4bf4-bba4-a703d294e815","Type":"ContainerStarted","Data":"809ee30d98bb3565ba5cf155951221a5005129295bb0d9bd140d1bb6d4d0f599"} Oct 31 14:01:44.232657 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:44.230673 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" event={"ID":"754e2b2d-0972-4bf4-bba4-a703d294e815","Type":"ContainerStarted","Data":"36261ae9e6e1172b9edb58dc77d4736eb8b8e4d54ba8d8e7c96c299e835572b9"} Oct 31 14:01:44.234966 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:44.234936 5532 scope.go:122] "RemoveContainer" containerID="21c4eb83f644f0d81a17c405c5d3144269e2945413cf830b2ebeb9460f367cd8" Oct 31 14:01:44.235284 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:44.235263 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c4eb83f644f0d81a17c405c5d3144269e2945413cf830b2ebeb9460f367cd8\": container with ID starting with 21c4eb83f644f0d81a17c405c5d3144269e2945413cf830b2ebeb9460f367cd8 not found: ID does not exist" containerID="21c4eb83f644f0d81a17c405c5d3144269e2945413cf830b2ebeb9460f367cd8" Oct 31 14:01:44.235358 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:44.235294 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c4eb83f644f0d81a17c405c5d3144269e2945413cf830b2ebeb9460f367cd8"} err="failed to get container status \"21c4eb83f644f0d81a17c405c5d3144269e2945413cf830b2ebeb9460f367cd8\": rpc error: code = NotFound desc = could not find container \"21c4eb83f644f0d81a17c405c5d3144269e2945413cf830b2ebeb9460f367cd8\": container with ID starting with 21c4eb83f644f0d81a17c405c5d3144269e2945413cf830b2ebeb9460f367cd8 not found: ID does not exist" Oct 31 14:01:44.245086 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:44.245049 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk"] Oct 31 14:01:44.252281 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:44.252257 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-974711-6c8dc687c9-bpnhk"] Oct 31 14:01:44.270882 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:44.270833 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" podStartSLOduration=2.270819245 podStartE2EDuration="2.270819245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:01:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:01:44.267610471 +0000 UTC m=+3472.413354294" watchObservedRunningTime="2026-10-31 14:01:44.270819245 +0000 UTC m=+3472.416563073" Oct 31 14:01:45.245072 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:45.245035 5532 generic.go:356] "Generic (PLEG): container finished" podID="754e2b2d-0972-4bf4-bba4-a703d294e815" containerID="809ee30d98bb3565ba5cf155951221a5005129295bb0d9bd140d1bb6d4d0f599" exitCode=0 Oct 31 14:01:45.245578 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:45.245122 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" event={"ID":"754e2b2d-0972-4bf4-bba4-a703d294e815","Type":"ContainerDied","Data":"809ee30d98bb3565ba5cf155951221a5005129295bb0d9bd140d1bb6d4d0f599"} Oct 31 14:01:45.617742 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:45.617510 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z"] Oct 31 14:01:45.626224 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:45.626193 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-33c10a-9sj2z"] Oct 31 14:01:46.021083 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:46.021040 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad79fea-9683-4d69-a99b-77e69db4bb68" path="/var/lib/kubelet/pods/8ad79fea-9683-4d69-a99b-77e69db4bb68/volumes" Oct 31 14:01:46.021810 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:46.021786 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3a4ae30-4d47-420b-bde7-c7174eabaedf" path="/var/lib/kubelet/pods/e3a4ae30-4d47-420b-bde7-c7174eabaedf/volumes" Oct 31 14:01:46.133121 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:46.133066 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" podUID="6cd8e2bd-93c5-4108-a69a-1648d856fede" containerName="payload-processing" containerID="cri-o://77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b" gracePeriod=25 Oct 31 14:01:46.407473 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:46.407446 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" Oct 31 14:01:46.459577 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:46.459542 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-openshift-ca\") pod \"754e2b2d-0972-4bf4-bba4-a703d294e815\" (UID: \"754e2b2d-0972-4bf4-bba4-a703d294e815\") " Oct 31 14:01:46.459775 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:46.459604 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/754e2b2d-0972-4bf4-bba4-a703d294e815-kube-api-access-xxq78\" (UniqueName: \"kubernetes.io/projected/754e2b2d-0972-4bf4-bba4-a703d294e815-kube-api-access-xxq78\") pod \"754e2b2d-0972-4bf4-bba4-a703d294e815\" (UID: \"754e2b2d-0972-4bf4-bba4-a703d294e815\") " Oct 31 14:01:46.459775 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:46.459634 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-xks-ca\") pod \"754e2b2d-0972-4bf4-bba4-a703d294e815\" (UID: \"754e2b2d-0972-4bf4-bba4-a703d294e815\") " Oct 31 14:01:46.460249 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:46.459974 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-openshift-ca" pod "754e2b2d-0972-4bf4-bba4-a703d294e815" (UID: "754e2b2d-0972-4bf4-bba4-a703d294e815"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:46.460249 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:46.460076 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:46.462103 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:46.462070 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-xks-ca" pod "754e2b2d-0972-4bf4-bba4-a703d294e815" (UID: "754e2b2d-0972-4bf4-bba4-a703d294e815"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:01:46.462628 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:46.462578 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/754e2b2d-0972-4bf4-bba4-a703d294e815-kube-api-access-xxq78" pod "754e2b2d-0972-4bf4-bba4-a703d294e815" (UID: "754e2b2d-0972-4bf4-bba4-a703d294e815"). InnerVolumeSpecName "kube-api-access-xxq78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:46.561505 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:46.561427 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxq78\" (UniqueName: \"kubernetes.io/projected/754e2b2d-0972-4bf4-bba4-a703d294e815-kube-api-access-xxq78\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:46.561505 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:46.561456 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/754e2b2d-0972-4bf4-bba4-a703d294e815-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:47.236061 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.236030 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 14:01:47.264141 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.264056 5532 generic.go:356] "Generic (PLEG): container finished" podID="f63ce148-84ae-4429-9661-77b0e688205a" containerID="c33cc72858300afe60d86c3981ceba0c38a044e698a8c2d41b308acde7e8a51c" exitCode=0 Oct 31 14:01:47.264344 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.264230 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" event={"ID":"f63ce148-84ae-4429-9661-77b0e688205a","Type":"ContainerDied","Data":"c33cc72858300afe60d86c3981ceba0c38a044e698a8c2d41b308acde7e8a51c"} Oct 31 14:01:47.264344 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.264268 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" event={"ID":"f63ce148-84ae-4429-9661-77b0e688205a","Type":"ContainerDied","Data":"95ce09366bfbd703905ae4ebcc8d024bac1b04fb953b730b4b8fe1004ee66658"} Oct 31 14:01:47.264344 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.264296 5532 scope.go:122] "RemoveContainer" containerID="c33cc72858300afe60d86c3981ceba0c38a044e698a8c2d41b308acde7e8a51c" Oct 31 14:01:47.264543 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.264374 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2" Oct 31 14:01:47.267523 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.267498 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" Oct 31 14:01:47.267772 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.267537 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd" event={"ID":"754e2b2d-0972-4bf4-bba4-a703d294e815","Type":"ContainerDied","Data":"36261ae9e6e1172b9edb58dc77d4736eb8b8e4d54ba8d8e7c96c299e835572b9"} Oct 31 14:01:47.267772 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.267585 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="36261ae9e6e1172b9edb58dc77d4736eb8b8e4d54ba8d8e7c96c299e835572b9" Oct 31 14:01:47.290837 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.290625 5532 scope.go:122] "RemoveContainer" containerID="9b373aaf8cbb57afa0a8f1b18d6cc3f77c8ac5b7ccd326f879b7f86f3afb8a20" Oct 31 14:01:47.302534 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.302500 5532 scope.go:122] "RemoveContainer" containerID="c33cc72858300afe60d86c3981ceba0c38a044e698a8c2d41b308acde7e8a51c" Oct 31 14:01:47.303012 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:47.302975 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c33cc72858300afe60d86c3981ceba0c38a044e698a8c2d41b308acde7e8a51c\": container with ID starting with c33cc72858300afe60d86c3981ceba0c38a044e698a8c2d41b308acde7e8a51c not found: ID does not exist" containerID="c33cc72858300afe60d86c3981ceba0c38a044e698a8c2d41b308acde7e8a51c" Oct 31 14:01:47.303128 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.303036 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c33cc72858300afe60d86c3981ceba0c38a044e698a8c2d41b308acde7e8a51c"} err="failed to get container status \"c33cc72858300afe60d86c3981ceba0c38a044e698a8c2d41b308acde7e8a51c\": rpc error: code = NotFound desc = could not find container \"c33cc72858300afe60d86c3981ceba0c38a044e698a8c2d41b308acde7e8a51c\": container with ID starting with c33cc72858300afe60d86c3981ceba0c38a044e698a8c2d41b308acde7e8a51c not found: ID does not exist" Oct 31 14:01:47.303128 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.303078 5532 scope.go:122] "RemoveContainer" containerID="9b373aaf8cbb57afa0a8f1b18d6cc3f77c8ac5b7ccd326f879b7f86f3afb8a20" Oct 31 14:01:47.303430 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:47.303407 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b373aaf8cbb57afa0a8f1b18d6cc3f77c8ac5b7ccd326f879b7f86f3afb8a20\": container with ID starting with 9b373aaf8cbb57afa0a8f1b18d6cc3f77c8ac5b7ccd326f879b7f86f3afb8a20 not found: ID does not exist" containerID="9b373aaf8cbb57afa0a8f1b18d6cc3f77c8ac5b7ccd326f879b7f86f3afb8a20" Oct 31 14:01:47.303554 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.303432 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b373aaf8cbb57afa0a8f1b18d6cc3f77c8ac5b7ccd326f879b7f86f3afb8a20"} err="failed to get container status \"9b373aaf8cbb57afa0a8f1b18d6cc3f77c8ac5b7ccd326f879b7f86f3afb8a20\": rpc error: code = NotFound desc = could not find container \"9b373aaf8cbb57afa0a8f1b18d6cc3f77c8ac5b7ccd326f879b7f86f3afb8a20\": container with ID starting with 9b373aaf8cbb57afa0a8f1b18d6cc3f77c8ac5b7ccd326f879b7f86f3afb8a20 not found: ID does not exist" Oct 31 14:01:47.390587 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.390554 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f63ce148-84ae-4429-9661-77b0e688205a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f63ce148-84ae-4429-9661-77b0e688205a-istio-podinfo\") pod \"f63ce148-84ae-4429-9661-77b0e688205a\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " Oct 31 14:01:47.390738 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.390639 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-data\") pod \"f63ce148-84ae-4429-9661-77b0e688205a\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " Oct 31 14:01:47.390952 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.390927 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-data" pod "f63ce148-84ae-4429-9661-77b0e688205a" (UID: "f63ce148-84ae-4429-9661-77b0e688205a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:47.391055 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.391033 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-credential-socket\") pod \"f63ce148-84ae-4429-9661-77b0e688205a\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " Oct 31 14:01:47.391231 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.391090 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-certs\") pod \"f63ce148-84ae-4429-9661-77b0e688205a\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " Oct 31 14:01:47.391340 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.391275 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-credential-socket" pod "f63ce148-84ae-4429-9661-77b0e688205a" (UID: "f63ce148-84ae-4429-9661-77b0e688205a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:47.391340 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.391282 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f63ce148-84ae-4429-9661-77b0e688205a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f63ce148-84ae-4429-9661-77b0e688205a-istiod-ca-cert\") pod \"f63ce148-84ae-4429-9661-77b0e688205a\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " Oct 31 14:01:47.391557 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.391403 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-istio-token\" (UniqueName: \"kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-istio-token\") pod \"f63ce148-84ae-4429-9661-77b0e688205a\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " Oct 31 14:01:47.391557 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.391459 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-certs" pod "f63ce148-84ae-4429-9661-77b0e688205a" (UID: "f63ce148-84ae-4429-9661-77b0e688205a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:47.391694 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.391575 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f63ce148-84ae-4429-9661-77b0e688205a-istiod-ca-cert" pod "f63ce148-84ae-4429-9661-77b0e688205a" (UID: "f63ce148-84ae-4429-9661-77b0e688205a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:47.391795 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.391772 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-socket\") pod \"f63ce148-84ae-4429-9661-77b0e688205a\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " Oct 31 14:01:47.391869 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.391827 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-kube-api-access-cjvw7\" (UniqueName: \"kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-kube-api-access-cjvw7\") pod \"f63ce148-84ae-4429-9661-77b0e688205a\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " Oct 31 14:01:47.391938 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.391883 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-envoy\") pod \"f63ce148-84ae-4429-9661-77b0e688205a\" (UID: \"f63ce148-84ae-4429-9661-77b0e688205a\") " Oct 31 14:01:47.392359 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.392338 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-socket" pod "f63ce148-84ae-4429-9661-77b0e688205a" (UID: "f63ce148-84ae-4429-9661-77b0e688205a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:47.392544 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.392509 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f63ce148-84ae-4429-9661-77b0e688205a-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:47.392544 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.392537 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:47.392745 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.392550 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:47.392745 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.392562 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:47.393551 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.393506 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f63ce148-84ae-4429-9661-77b0e688205a-istio-podinfo" pod "f63ce148-84ae-4429-9661-77b0e688205a" (UID: "f63ce148-84ae-4429-9661-77b0e688205a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:01:47.398914 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.398878 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-istio-token" pod "f63ce148-84ae-4429-9661-77b0e688205a" (UID: "f63ce148-84ae-4429-9661-77b0e688205a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:47.399075 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.399057 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-envoy" pod "f63ce148-84ae-4429-9661-77b0e688205a" (UID: "f63ce148-84ae-4429-9661-77b0e688205a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:47.399453 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.399426 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-kube-api-access-cjvw7" pod "f63ce148-84ae-4429-9661-77b0e688205a" (UID: "f63ce148-84ae-4429-9661-77b0e688205a"). InnerVolumeSpecName "kube-api-access-cjvw7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:47.494165 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.494128 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:47.494165 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.494160 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:47.494165 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.494171 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjvw7\" (UniqueName: \"kubernetes.io/projected/f63ce148-84ae-4429-9661-77b0e688205a-kube-api-access-cjvw7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:47.494694 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.494181 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f63ce148-84ae-4429-9661-77b0e688205a-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:47.494694 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.494190 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f63ce148-84ae-4429-9661-77b0e688205a-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:47.593090 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.593054 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2"] Oct 31 14:01:47.603460 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:47.603431 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-974711-openshift-default-797ffd4df5-g57p2"] Oct 31 14:01:48.003900 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:48.003861 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f63ce148-84ae-4429-9661-77b0e688205a" path="/var/lib/kubelet/pods/f63ce148-84ae-4429-9661-77b0e688205a/volumes" Oct 31 14:01:50.972269 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:50.972225 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" containerID="cri-o://caf649ca72c724aa6644224ec71c71dfc184eb0db2943a1a2b94c24ecba5eecb" gracePeriod=45 Oct 31 14:01:50.989759 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:50.989720 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" containerID="cri-o://c2fad5b6b956521fc75065e7de8355394756b996e04f047b0bbe60155bc84639" gracePeriod=45 Oct 31 14:01:51.081791 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.081757 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" probeResult="failure" output="Get \"https://10.132.0.54:8000/ready\": dial tcp 10.132.0.54:8000: connect: connection refused" Oct 31 14:01:51.300092 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.300067 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 14:01:51.322204 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.322174 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 14:01:51.344314 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.344282 5532 generic.go:356] "Generic (PLEG): container finished" podID="69038604-7198-4e89-be50-1a155aaa7c75" containerID="c2fad5b6b956521fc75065e7de8355394756b996e04f047b0bbe60155bc84639" exitCode=0 Oct 31 14:01:51.344532 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.344340 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" Oct 31 14:01:51.344532 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.344373 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" event={"ID":"69038604-7198-4e89-be50-1a155aaa7c75","Type":"ContainerDied","Data":"c2fad5b6b956521fc75065e7de8355394756b996e04f047b0bbe60155bc84639"} Oct 31 14:01:51.344532 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.344421 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd" event={"ID":"69038604-7198-4e89-be50-1a155aaa7c75","Type":"ContainerDied","Data":"2ff0ab847721014708ecd1500baca02cf5c1ab77fb83787937f1809becd5e20d"} Oct 31 14:01:51.344532 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.344445 5532 scope.go:122] "RemoveContainer" containerID="c2fad5b6b956521fc75065e7de8355394756b996e04f047b0bbe60155bc84639" Oct 31 14:01:51.346550 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.346524 5532 generic.go:356] "Generic (PLEG): container finished" podID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerID="caf649ca72c724aa6644224ec71c71dfc184eb0db2943a1a2b94c24ecba5eecb" exitCode=0 Oct 31 14:01:51.346693 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.346654 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" Oct 31 14:01:51.346774 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.346687 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" event={"ID":"1e02db96-5a0a-4bca-812e-af51948d7ab0","Type":"ContainerDied","Data":"caf649ca72c724aa6644224ec71c71dfc184eb0db2943a1a2b94c24ecba5eecb"} Oct 31 14:01:51.346774 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.346722 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q" event={"ID":"1e02db96-5a0a-4bca-812e-af51948d7ab0","Type":"ContainerDied","Data":"779b2bbf105441b2d460151061cc4f08f0933d90e3e49e6faf763d81b41113c2"} Oct 31 14:01:51.361579 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.361544 5532 scope.go:122] "RemoveContainer" containerID="c2fad5b6b956521fc75065e7de8355394756b996e04f047b0bbe60155bc84639" Oct 31 14:01:51.362004 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:51.361962 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2fad5b6b956521fc75065e7de8355394756b996e04f047b0bbe60155bc84639\": container with ID starting with c2fad5b6b956521fc75065e7de8355394756b996e04f047b0bbe60155bc84639 not found: ID does not exist" containerID="c2fad5b6b956521fc75065e7de8355394756b996e04f047b0bbe60155bc84639" Oct 31 14:01:51.362105 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.362015 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2fad5b6b956521fc75065e7de8355394756b996e04f047b0bbe60155bc84639"} err="failed to get container status \"c2fad5b6b956521fc75065e7de8355394756b996e04f047b0bbe60155bc84639\": rpc error: code = NotFound desc = could not find container \"c2fad5b6b956521fc75065e7de8355394756b996e04f047b0bbe60155bc84639\": container with ID starting with c2fad5b6b956521fc75065e7de8355394756b996e04f047b0bbe60155bc84639 not found: ID does not exist" Oct 31 14:01:51.362105 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.362033 5532 scope.go:122] "RemoveContainer" containerID="caf649ca72c724aa6644224ec71c71dfc184eb0db2943a1a2b94c24ecba5eecb" Oct 31 14:01:51.380174 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.380136 5532 scope.go:122] "RemoveContainer" containerID="caf649ca72c724aa6644224ec71c71dfc184eb0db2943a1a2b94c24ecba5eecb" Oct 31 14:01:51.380529 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:51.380504 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf649ca72c724aa6644224ec71c71dfc184eb0db2943a1a2b94c24ecba5eecb\": container with ID starting with caf649ca72c724aa6644224ec71c71dfc184eb0db2943a1a2b94c24ecba5eecb not found: ID does not exist" containerID="caf649ca72c724aa6644224ec71c71dfc184eb0db2943a1a2b94c24ecba5eecb" Oct 31 14:01:51.380657 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.380542 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf649ca72c724aa6644224ec71c71dfc184eb0db2943a1a2b94c24ecba5eecb"} err="failed to get container status \"caf649ca72c724aa6644224ec71c71dfc184eb0db2943a1a2b94c24ecba5eecb\": rpc error: code = NotFound desc = could not find container \"caf649ca72c724aa6644224ec71c71dfc184eb0db2943a1a2b94c24ecba5eecb\": container with ID starting with caf649ca72c724aa6644224ec71c71dfc184eb0db2943a1a2b94c24ecba5eecb not found: ID does not exist" Oct 31 14:01:51.458880 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.458838 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-home\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-home\") pod \"1e02db96-5a0a-4bca-812e-af51948d7ab0\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " Oct 31 14:01:51.459133 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.458909 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-dshm\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-dshm\") pod \"69038604-7198-4e89-be50-1a155aaa7c75\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " Oct 31 14:01:51.459133 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.458938 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-tmp-dir\") pod \"1e02db96-5a0a-4bca-812e-af51948d7ab0\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " Oct 31 14:01:51.459133 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.458976 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69038604-7198-4e89-be50-1a155aaa7c75-tls-certs\" (UniqueName: \"kubernetes.io/secret/69038604-7198-4e89-be50-1a155aaa7c75-tls-certs\") pod \"69038604-7198-4e89-be50-1a155aaa7c75\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " Oct 31 14:01:51.459133 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.459005 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e02db96-5a0a-4bca-812e-af51948d7ab0-kube-api-access-sh7l5\" (UniqueName: \"kubernetes.io/projected/1e02db96-5a0a-4bca-812e-af51948d7ab0-kube-api-access-sh7l5\") pod \"1e02db96-5a0a-4bca-812e-af51948d7ab0\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " Oct 31 14:01:51.459133 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.459075 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e02db96-5a0a-4bca-812e-af51948d7ab0-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e02db96-5a0a-4bca-812e-af51948d7ab0-tls-certs\") pod \"1e02db96-5a0a-4bca-812e-af51948d7ab0\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " Oct 31 14:01:51.459133 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.459100 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-model-cache\") pod \"1e02db96-5a0a-4bca-812e-af51948d7ab0\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " Oct 31 14:01:51.459722 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.459256 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-tmp-dir\") pod \"69038604-7198-4e89-be50-1a155aaa7c75\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " Oct 31 14:01:51.459722 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.459310 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69038604-7198-4e89-be50-1a155aaa7c75-kube-api-access-5vwhg\" (UniqueName: \"kubernetes.io/projected/69038604-7198-4e89-be50-1a155aaa7c75-kube-api-access-5vwhg\") pod \"69038604-7198-4e89-be50-1a155aaa7c75\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " Oct 31 14:01:51.459722 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.459357 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-home\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-home\") pod \"69038604-7198-4e89-be50-1a155aaa7c75\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " Oct 31 14:01:51.459722 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.459376 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-home" pod "1e02db96-5a0a-4bca-812e-af51948d7ab0" (UID: "1e02db96-5a0a-4bca-812e-af51948d7ab0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:51.459722 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.459530 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-dshm\") pod \"1e02db96-5a0a-4bca-812e-af51948d7ab0\" (UID: \"1e02db96-5a0a-4bca-812e-af51948d7ab0\") " Oct 31 14:01:51.459722 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.459561 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-model-cache" pod "1e02db96-5a0a-4bca-812e-af51948d7ab0" (UID: "1e02db96-5a0a-4bca-812e-af51948d7ab0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:51.459722 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.459576 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-model-cache\") pod \"69038604-7198-4e89-be50-1a155aaa7c75\" (UID: \"69038604-7198-4e89-be50-1a155aaa7c75\") " Oct 31 14:01:51.460120 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.459891 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-tmp-dir" pod "1e02db96-5a0a-4bca-812e-af51948d7ab0" (UID: "1e02db96-5a0a-4bca-812e-af51948d7ab0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:51.460184 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.460129 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-model-cache" pod "69038604-7198-4e89-be50-1a155aaa7c75" (UID: "69038604-7198-4e89-be50-1a155aaa7c75"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:51.460411 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.460389 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:51.460509 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.460417 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:51.460509 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.460432 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:51.460509 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.460445 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:51.460509 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.460150 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-home" pod "69038604-7198-4e89-be50-1a155aaa7c75" (UID: "69038604-7198-4e89-be50-1a155aaa7c75"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:51.464429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.462997 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-tmp-dir" pod "69038604-7198-4e89-be50-1a155aaa7c75" (UID: "69038604-7198-4e89-be50-1a155aaa7c75"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:51.465821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.465787 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e02db96-5a0a-4bca-812e-af51948d7ab0-kube-api-access-sh7l5" pod "1e02db96-5a0a-4bca-812e-af51948d7ab0" (UID: "1e02db96-5a0a-4bca-812e-af51948d7ab0"). InnerVolumeSpecName "kube-api-access-sh7l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:51.466382 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.466337 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-dshm" pod "1e02db96-5a0a-4bca-812e-af51948d7ab0" (UID: "1e02db96-5a0a-4bca-812e-af51948d7ab0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:51.466480 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.466388 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69038604-7198-4e89-be50-1a155aaa7c75-tls-certs" pod "69038604-7198-4e89-be50-1a155aaa7c75" (UID: "69038604-7198-4e89-be50-1a155aaa7c75"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:01:51.466535 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.466518 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69038604-7198-4e89-be50-1a155aaa7c75-kube-api-access-5vwhg" pod "69038604-7198-4e89-be50-1a155aaa7c75" (UID: "69038604-7198-4e89-be50-1a155aaa7c75"). InnerVolumeSpecName "kube-api-access-5vwhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:51.466738 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.466717 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e02db96-5a0a-4bca-812e-af51948d7ab0-tls-certs" pod "1e02db96-5a0a-4bca-812e-af51948d7ab0" (UID: "1e02db96-5a0a-4bca-812e-af51948d7ab0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:01:51.467024 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.467005 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-dshm" pod "69038604-7198-4e89-be50-1a155aaa7c75" (UID: "69038604-7198-4e89-be50-1a155aaa7c75"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:51.561963 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.561883 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:51.561963 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.561912 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vwhg\" (UniqueName: \"kubernetes.io/projected/69038604-7198-4e89-be50-1a155aaa7c75-kube-api-access-5vwhg\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:51.561963 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.561922 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:51.561963 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.561930 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e02db96-5a0a-4bca-812e-af51948d7ab0-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:51.561963 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.561938 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69038604-7198-4e89-be50-1a155aaa7c75-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:51.561963 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.561945 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69038604-7198-4e89-be50-1a155aaa7c75-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:51.561963 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.561957 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh7l5\" (UniqueName: \"kubernetes.io/projected/1e02db96-5a0a-4bca-812e-af51948d7ab0-kube-api-access-sh7l5\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:51.561963 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.561965 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e02db96-5a0a-4bca-812e-af51948d7ab0-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:51.676999 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.676971 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd"] Oct 31 14:01:51.683818 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.683793 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/simulator-w1-d85d82-kserve-6dddf5465-8nntd"] Oct 31 14:01:51.696701 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.696674 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" podUID="de6b3aca-3802-48c4-9315-2492f01492a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:01:51.701457 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.701434 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q"] Oct 31 14:01:51.707362 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:51.707340 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-d85d82/premium-w1-d85d82-kserve-d5d69db44-wt98q"] Oct 31 14:01:52.003209 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:52.003171 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" path="/var/lib/kubelet/pods/1e02db96-5a0a-4bca-812e-af51948d7ab0/volumes" Oct 31 14:01:52.003675 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:52.003660 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69038604-7198-4e89-be50-1a155aaa7c75" path="/var/lib/kubelet/pods/69038604-7198-4e89-be50-1a155aaa7c75/volumes" Oct 31 14:01:52.981021 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:52.980985 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr"] Oct 31 14:01:53.790220 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:53.790185 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" podUID="6cd8e2bd-93c5-4108-a69a-1648d856fede" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:01:54.126613 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:54.126510 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r"] Oct 31 14:01:54.126809 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:54.126779 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" podUID="53e8516d-253e-454e-b928-c24938cfe79b" containerName="istio-proxy" containerID="cri-o://bc44cb99387738a539634d9154a08d357b665467d3fcd59a713bc5379bcc9551" gracePeriod=30 Oct 31 14:01:56.472421 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:56.472375 5532 scope.go:122] "RemoveContainer" containerID="89d55aefc5feb53d811a83c9397abb0ba55c5531977e5f177e9c4a1681aaa6e1" Oct 31 14:01:56.482841 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:56.482819 5532 scope.go:122] "RemoveContainer" containerID="dc05081c75714d4a08958c35a135bd5b7657fe584b82c004c55db7a21216498d" Oct 31 14:01:56.493621 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:56.493587 5532 scope.go:122] "RemoveContainer" containerID="003db0eb79a064442fecbd645922acaedcc9c9694b67a92baf925022b380dc94" Oct 31 14:01:56.506363 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:56.506344 5532 scope.go:122] "RemoveContainer" containerID="bc960d18b7627f3fb039dd5864d0150436dbab5cb1f8b8af0733939907189c1b" Oct 31 14:01:56.515748 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:56.515721 5532 scope.go:122] "RemoveContainer" containerID="c577b7ca0c4fc628b263254d3f09a1d2e3cebfea8543b5472cb1b94f4a2a66e3" Oct 31 14:01:56.526697 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:56.526681 5532 scope.go:122] "RemoveContainer" containerID="b9cb75f4ce6ececaf3641da8b00e079c6475cee761e875908be40b18f5a5dc84" Oct 31 14:01:56.535679 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:56.535664 5532 scope.go:122] "RemoveContainer" containerID="7eaa827be8b760e6c3bbda47c22d9e291d72c2f84b616ec7b75a9c55a7a8066c" Oct 31 14:01:57.981369 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:57.981324 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" podUID="b45f1106-92ed-4bad-b7c0-aa3a0442793b" containerName="payload-processing" containerID="cri-o://011936b68cb4fbbd96119431b3062007e1fda2275ecf43592ca1381f6c33f015" gracePeriod=25 Oct 31 14:01:58.906779 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:58.906741 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" podUID="b45f1106-92ed-4bad-b7c0-aa3a0442793b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:01:59.282037 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.281980 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd"] Oct 31 14:01:59.285515 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.285484 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-15dc2f-fr9xd"] Oct 31 14:01:59.410184 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.410110 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 14:01:59.427283 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.427256 5532 generic.go:356] "Generic (PLEG): container finished" podID="53e8516d-253e-454e-b928-c24938cfe79b" containerID="bc44cb99387738a539634d9154a08d357b665467d3fcd59a713bc5379bcc9551" exitCode=0 Oct 31 14:01:59.427467 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.427312 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" Oct 31 14:01:59.427467 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.427340 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" event={"ID":"53e8516d-253e-454e-b928-c24938cfe79b","Type":"ContainerDied","Data":"bc44cb99387738a539634d9154a08d357b665467d3fcd59a713bc5379bcc9551"} Oct 31 14:01:59.427467 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.427396 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r" event={"ID":"53e8516d-253e-454e-b928-c24938cfe79b","Type":"ContainerDied","Data":"448705a2ae6113ffacca5f271f323d94f3083ebb62790a912702e3ce40e58421"} Oct 31 14:01:59.427467 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.427418 5532 scope.go:122] "RemoveContainer" containerID="bc44cb99387738a539634d9154a08d357b665467d3fcd59a713bc5379bcc9551" Oct 31 14:01:59.438115 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.438100 5532 scope.go:122] "RemoveContainer" containerID="bc44cb99387738a539634d9154a08d357b665467d3fcd59a713bc5379bcc9551" Oct 31 14:01:59.438376 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:01:59.438356 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc44cb99387738a539634d9154a08d357b665467d3fcd59a713bc5379bcc9551\": container with ID starting with bc44cb99387738a539634d9154a08d357b665467d3fcd59a713bc5379bcc9551 not found: ID does not exist" containerID="bc44cb99387738a539634d9154a08d357b665467d3fcd59a713bc5379bcc9551" Oct 31 14:01:59.438452 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.438384 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc44cb99387738a539634d9154a08d357b665467d3fcd59a713bc5379bcc9551"} err="failed to get container status \"bc44cb99387738a539634d9154a08d357b665467d3fcd59a713bc5379bcc9551\": rpc error: code = NotFound desc = could not find container \"bc44cb99387738a539634d9154a08d357b665467d3fcd59a713bc5379bcc9551\": container with ID starting with bc44cb99387738a539634d9154a08d357b665467d3fcd59a713bc5379bcc9551 not found: ID does not exist" Oct 31 14:01:59.460115 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.460084 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-kube-api-access-758dw\" (UniqueName: \"kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-kube-api-access-758dw\") pod \"53e8516d-253e-454e-b928-c24938cfe79b\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " Oct 31 14:01:59.460246 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.460186 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-envoy\") pod \"53e8516d-253e-454e-b928-c24938cfe79b\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " Oct 31 14:01:59.460246 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.460233 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-socket\") pod \"53e8516d-253e-454e-b928-c24938cfe79b\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " Oct 31 14:01:59.460331 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.460287 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53e8516d-253e-454e-b928-c24938cfe79b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e8516d-253e-454e-b928-c24938cfe79b-istiod-ca-cert\") pod \"53e8516d-253e-454e-b928-c24938cfe79b\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " Oct 31 14:01:59.460377 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.460341 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-credential-socket\") pod \"53e8516d-253e-454e-b928-c24938cfe79b\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " Oct 31 14:01:59.460513 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.460484 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53e8516d-253e-454e-b928-c24938cfe79b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e8516d-253e-454e-b928-c24938cfe79b-istio-podinfo\") pod \"53e8516d-253e-454e-b928-c24938cfe79b\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " Oct 31 14:01:59.460626 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.460569 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-istio-token\" (UniqueName: \"kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-istio-token\") pod \"53e8516d-253e-454e-b928-c24938cfe79b\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " Oct 31 14:01:59.460626 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.460585 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-socket" pod "53e8516d-253e-454e-b928-c24938cfe79b" (UID: "53e8516d-253e-454e-b928-c24938cfe79b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:59.460772 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.460633 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-credential-socket" pod "53e8516d-253e-454e-b928-c24938cfe79b" (UID: "53e8516d-253e-454e-b928-c24938cfe79b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:59.460772 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.460643 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-data\") pod \"53e8516d-253e-454e-b928-c24938cfe79b\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " Oct 31 14:01:59.460772 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.460719 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-certs\") pod \"53e8516d-253e-454e-b928-c24938cfe79b\" (UID: \"53e8516d-253e-454e-b928-c24938cfe79b\") " Oct 31 14:01:59.460772 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.460722 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53e8516d-253e-454e-b928-c24938cfe79b-istiod-ca-cert" pod "53e8516d-253e-454e-b928-c24938cfe79b" (UID: "53e8516d-253e-454e-b928-c24938cfe79b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:01:59.461045 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.461000 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-data" pod "53e8516d-253e-454e-b928-c24938cfe79b" (UID: "53e8516d-253e-454e-b928-c24938cfe79b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:59.461130 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.461106 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-certs" pod "53e8516d-253e-454e-b928-c24938cfe79b" (UID: "53e8516d-253e-454e-b928-c24938cfe79b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:59.461531 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.461505 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:59.461724 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.461535 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53e8516d-253e-454e-b928-c24938cfe79b-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:59.461724 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.461551 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:59.461724 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.461568 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:59.461724 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.461582 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:59.462862 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.462840 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-envoy" pod "53e8516d-253e-454e-b928-c24938cfe79b" (UID: "53e8516d-253e-454e-b928-c24938cfe79b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:01:59.462862 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.462850 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-kube-api-access-758dw" pod "53e8516d-253e-454e-b928-c24938cfe79b" (UID: "53e8516d-253e-454e-b928-c24938cfe79b"). InnerVolumeSpecName "kube-api-access-758dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:59.463102 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.463080 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-istio-token" pod "53e8516d-253e-454e-b928-c24938cfe79b" (UID: "53e8516d-253e-454e-b928-c24938cfe79b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:01:59.463317 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.463301 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53e8516d-253e-454e-b928-c24938cfe79b-istio-podinfo" pod "53e8516d-253e-454e-b928-c24938cfe79b" (UID: "53e8516d-253e-454e-b928-c24938cfe79b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:01:59.566503 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.566399 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53e8516d-253e-454e-b928-c24938cfe79b-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:59.566503 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.566435 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:59.566503 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.566448 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-758dw\" (UniqueName: \"kubernetes.io/projected/53e8516d-253e-454e-b928-c24938cfe79b-kube-api-access-758dw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:59.566503 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.566463 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53e8516d-253e-454e-b928-c24938cfe79b-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:01:59.756012 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.755974 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r"] Oct 31 14:01:59.758965 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:01:59.758940 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-33c10a-openshift-default-b667dc5bc-g4g8r"] Oct 31 14:02:00.004142 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:00.004109 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53e8516d-253e-454e-b928-c24938cfe79b" path="/var/lib/kubelet/pods/53e8516d-253e-454e-b928-c24938cfe79b/volumes" Oct 31 14:02:00.004621 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:00.004588 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="754e2b2d-0972-4bf4-bba4-a703d294e815" path="/var/lib/kubelet/pods/754e2b2d-0972-4bf4-bba4-a703d294e815/volumes" Oct 31 14:02:01.248857 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.248822 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj"] Oct 31 14:02:01.249536 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249517 5532 cpu_manager.go:414] "containerMap: removing container" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" Oct 31 14:02:01.249615 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249538 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" Oct 31 14:02:01.249615 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249551 5532 cpu_manager.go:414] "containerMap: removing container" podUID="8ad79fea-9683-4d69-a99b-77e69db4bb68" containerName="payload-pre-processing" Oct 31 14:02:01.249615 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249556 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad79fea-9683-4d69-a99b-77e69db4bb68" containerName="payload-pre-processing" Oct 31 14:02:01.249615 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249564 5532 cpu_manager.go:414] "containerMap: removing container" podUID="53e8516d-253e-454e-b928-c24938cfe79b" containerName="istio-proxy" Oct 31 14:02:01.249615 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249569 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53e8516d-253e-454e-b928-c24938cfe79b" containerName="istio-proxy" Oct 31 14:02:01.249615 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249581 5532 cpu_manager.go:414] "containerMap: removing container" podUID="754e2b2d-0972-4bf4-bba4-a703d294e815" containerName="revoke-keys" Oct 31 14:02:01.249615 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249586 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="754e2b2d-0972-4bf4-bba4-a703d294e815" containerName="revoke-keys" Oct 31 14:02:01.249615 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249606 5532 cpu_manager.go:414] "containerMap: removing container" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" Oct 31 14:02:01.249615 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249614 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" Oct 31 14:02:01.249923 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249623 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" Oct 31 14:02:01.249923 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249630 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" Oct 31 14:02:01.249923 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249647 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" Oct 31 14:02:01.249923 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249652 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" Oct 31 14:02:01.249923 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249758 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="754e2b2d-0972-4bf4-bba4-a703d294e815" containerName="revoke-keys" Oct 31 14:02:01.249923 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249768 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="69038604-7198-4e89-be50-1a155aaa7c75" containerName="main" Oct 31 14:02:01.249923 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249775 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" Oct 31 14:02:01.249923 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249789 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e02db96-5a0a-4bca-812e-af51948d7ab0" containerName="main" Oct 31 14:02:01.249923 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249795 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad79fea-9683-4d69-a99b-77e69db4bb68" containerName="payload-pre-processing" Oct 31 14:02:01.249923 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.249800 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="53e8516d-253e-454e-b928-c24938cfe79b" containerName="istio-proxy" Oct 31 14:02:01.254666 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.254644 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.267391 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.267366 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj"] Oct 31 14:02:01.292530 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.292496 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-credential-socket\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.292729 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.292551 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-envoy\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.292729 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.292668 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istiod-ca-cert\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.292729 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.292712 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-podinfo\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.292901 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.292806 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5gm5\" (UniqueName: \"kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-kube-api-access-m5gm5\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.292958 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.292888 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-socket\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.293041 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.293019 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-certs\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.293094 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.293053 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-token\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.293169 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.293149 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-data\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.317552 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.317524 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1b75f9-openshift-default-dockercfg-ln76p\"" Oct 31 14:02:01.395065 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.395023 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-certs\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.395252 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.395072 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-token\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.395252 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.395106 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-data\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.395350 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.395267 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-credential-socket\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.395350 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.395319 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-envoy\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.395433 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.395354 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istiod-ca-cert\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.395433 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.395391 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-podinfo\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.395518 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.395451 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5gm5\" (UniqueName: \"kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-kube-api-access-m5gm5\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.395518 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.395487 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-socket\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.396019 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.395995 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-socket\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.396278 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.396259 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-certs\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.396880 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.396849 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-data\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.397117 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.397098 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-credential-socket\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.399771 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.399739 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-envoy\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.399771 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.399763 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-podinfo\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.400001 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.399978 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istiod-ca-cert\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.410623 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.410582 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-token\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.413691 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.413661 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5gm5\" (UniqueName: \"kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-kube-api-access-m5gm5\") pod \"e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.570922 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.570825 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:01.585160 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.585136 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:02:01.695382 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:02:01.695359 5532 logging.go:55] [core] [Channel #10875 SubChannel #10876]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.139:50052", ServerName: "10.132.0.139:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.139:50052: connect: connection refused" Oct 31 14:02:01.762519 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:01.762485 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:02:01.862276 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:02:01.862213 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca77a1a_0c6a_4264_bc01_4899ef1dad75.slice/crio-c8a4d25183d37b534c3a4ba74410ad794769dd74fbe9db3200088abef37a6cd1 WatchSource:0}: Error finding container c8a4d25183d37b534c3a4ba74410ad794769dd74fbe9db3200088abef37a6cd1: Status 404 returned error can't find the container with id c8a4d25183d37b534c3a4ba74410ad794769dd74fbe9db3200088abef37a6cd1 Oct 31 14:02:01.864893 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.864225 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj"] Oct 31 14:02:01.866717 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.866678 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:02:01.866812 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.866768 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:02:01.866870 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.866809 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:02:01.940268 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:01.940247 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" Oct 31 14:02:02.011151 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.011116 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de6b3aca-3802-48c4-9315-2492f01492a0-kube-api-access-4c9c6\" (UniqueName: \"kubernetes.io/projected/de6b3aca-3802-48c4-9315-2492f01492a0-kube-api-access-4c9c6\") pod \"de6b3aca-3802-48c4-9315-2492f01492a0\" (UID: \"de6b3aca-3802-48c4-9315-2492f01492a0\") " Oct 31 14:02:02.011381 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.011313 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de6b3aca-3802-48c4-9315-2492f01492a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6b3aca-3802-48c4-9315-2492f01492a0-plugins-config-volume\") pod \"de6b3aca-3802-48c4-9315-2492f01492a0\" (UID: \"de6b3aca-3802-48c4-9315-2492f01492a0\") " Oct 31 14:02:02.013731 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.013701 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de6b3aca-3802-48c4-9315-2492f01492a0-plugins-config-volume" pod "de6b3aca-3802-48c4-9315-2492f01492a0" (UID: "de6b3aca-3802-48c4-9315-2492f01492a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:02.027286 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.027246 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6b3aca-3802-48c4-9315-2492f01492a0-kube-api-access-4c9c6" pod "de6b3aca-3802-48c4-9315-2492f01492a0" (UID: "de6b3aca-3802-48c4-9315-2492f01492a0"). InnerVolumeSpecName "kube-api-access-4c9c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:02.114854 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.114774 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6b3aca-3802-48c4-9315-2492f01492a0-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:02.114854 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.114803 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c9c6\" (UniqueName: \"kubernetes.io/projected/de6b3aca-3802-48c4-9315-2492f01492a0-kube-api-access-4c9c6\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:02.455849 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.455805 5532 generic.go:356] "Generic (PLEG): container finished" podID="de6b3aca-3802-48c4-9315-2492f01492a0" containerID="7ba608848fc44cfe24b5895340fb2dacd05f9085ac4ab9130847d3cd6fff3511" exitCode=0 Oct 31 14:02:02.456342 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.455891 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" Oct 31 14:02:02.456342 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.456057 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" event={"ID":"de6b3aca-3802-48c4-9315-2492f01492a0","Type":"ContainerDied","Data":"7ba608848fc44cfe24b5895340fb2dacd05f9085ac4ab9130847d3cd6fff3511"} Oct 31 14:02:02.456342 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.456112 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" event={"ID":"de6b3aca-3802-48c4-9315-2492f01492a0","Type":"ContainerDied","Data":"506215e8a6e912a860a5eacbe37372326739da6acd61c71019c682e9db0a6b6e"} Oct 31 14:02:02.456342 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.456138 5532 scope.go:122] "RemoveContainer" containerID="7ba608848fc44cfe24b5895340fb2dacd05f9085ac4ab9130847d3cd6fff3511" Oct 31 14:02:02.459307 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.459280 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" event={"ID":"6ca77a1a-0c6a-4264-bc01-4899ef1dad75","Type":"ContainerStarted","Data":"e2723632e4d3eb942d4650493f817764fc0f9bf7430f4cc4bbb3ea2d5feed0bb"} Oct 31 14:02:02.459532 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.459512 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" event={"ID":"6ca77a1a-0c6a-4264-bc01-4899ef1dad75","Type":"ContainerStarted","Data":"c8a4d25183d37b534c3a4ba74410ad794769dd74fbe9db3200088abef37a6cd1"} Oct 31 14:02:02.469377 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.469328 5532 scope.go:122] "RemoveContainer" containerID="7ba608848fc44cfe24b5895340fb2dacd05f9085ac4ab9130847d3cd6fff3511" Oct 31 14:02:02.470196 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:02.470168 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba608848fc44cfe24b5895340fb2dacd05f9085ac4ab9130847d3cd6fff3511\": container with ID starting with 7ba608848fc44cfe24b5895340fb2dacd05f9085ac4ab9130847d3cd6fff3511 not found: ID does not exist" containerID="7ba608848fc44cfe24b5895340fb2dacd05f9085ac4ab9130847d3cd6fff3511" Oct 31 14:02:02.470196 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.470205 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba608848fc44cfe24b5895340fb2dacd05f9085ac4ab9130847d3cd6fff3511"} err="failed to get container status \"7ba608848fc44cfe24b5895340fb2dacd05f9085ac4ab9130847d3cd6fff3511\": rpc error: code = NotFound desc = could not find container \"7ba608848fc44cfe24b5895340fb2dacd05f9085ac4ab9130847d3cd6fff3511\": container with ID starting with 7ba608848fc44cfe24b5895340fb2dacd05f9085ac4ab9130847d3cd6fff3511 not found: ID does not exist" Oct 31 14:02:02.493589 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.493547 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" podStartSLOduration=1.4935343589999999 podStartE2EDuration="1.493534359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:02.488784243 +0000 UTC m=+3490.634528071" watchObservedRunningTime="2026-10-31 14:02:02.493534359 +0000 UTC m=+3490.639278187" Oct 31 14:02:02.506724 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.506688 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f"] Oct 31 14:02:02.512648 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.512623 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f"] Oct 31 14:02:02.571029 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.570997 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:02.576514 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.576489 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:02.695343 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:02.695252 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-33c10a-7bd6bddc85-h5w8f" podUID="de6b3aca-3802-48c4-9315-2492f01492a0" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.139:50052\" within 1s: context deadline exceeded" Oct 31 14:02:03.468966 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:03.468933 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:03.470032 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:03.470014 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:03.788801 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:03.788708 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" podUID="6cd8e2bd-93c5-4108-a69a-1648d856fede" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:02:04.010074 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.010035 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de6b3aca-3802-48c4-9315-2492f01492a0" path="/var/lib/kubelet/pods/de6b3aca-3802-48c4-9315-2492f01492a0/volumes" Oct 31 14:02:04.010500 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.010483 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b"] Oct 31 14:02:04.010764 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.010741 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" podUID="4ab49081-96ed-405c-80d9-846b102649d7" containerName="maas-api" containerID="cri-o://db43910dc7ea528993d2ade2b37707d28815cefe512a5083c0ce66c4ecc9e70d" gracePeriod=30 Oct 31 14:02:04.339067 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.339041 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 14:02:04.472678 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.472644 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ab49081-96ed-405c-80d9-846b102649d7-kube-api-access-r8qbr\" (UniqueName: \"kubernetes.io/projected/4ab49081-96ed-405c-80d9-846b102649d7-kube-api-access-r8qbr\") pod \"4ab49081-96ed-405c-80d9-846b102649d7\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " Oct 31 14:02:04.473205 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.472713 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-maas-api-tls\") pod \"4ab49081-96ed-405c-80d9-846b102649d7\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " Oct 31 14:02:04.473205 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.472767 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-metrics-certs\") pod \"4ab49081-96ed-405c-80d9-846b102649d7\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " Oct 31 14:02:04.473205 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.472935 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ab49081-96ed-405c-80d9-846b102649d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab49081-96ed-405c-80d9-846b102649d7-openshift-service-ca\") pod \"4ab49081-96ed-405c-80d9-846b102649d7\" (UID: \"4ab49081-96ed-405c-80d9-846b102649d7\") " Oct 31 14:02:04.473384 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.473337 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ab49081-96ed-405c-80d9-846b102649d7-openshift-service-ca" pod "4ab49081-96ed-405c-80d9-846b102649d7" (UID: "4ab49081-96ed-405c-80d9-846b102649d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:04.473583 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.473561 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ab49081-96ed-405c-80d9-846b102649d7-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:04.475046 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.475021 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-maas-api-tls" pod "4ab49081-96ed-405c-80d9-846b102649d7" (UID: "4ab49081-96ed-405c-80d9-846b102649d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:04.475146 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.475044 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ab49081-96ed-405c-80d9-846b102649d7-kube-api-access-r8qbr" pod "4ab49081-96ed-405c-80d9-846b102649d7" (UID: "4ab49081-96ed-405c-80d9-846b102649d7"). InnerVolumeSpecName "kube-api-access-r8qbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:04.475383 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.475361 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-metrics-certs" pod "4ab49081-96ed-405c-80d9-846b102649d7" (UID: "4ab49081-96ed-405c-80d9-846b102649d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:04.478628 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.478588 5532 generic.go:356] "Generic (PLEG): container finished" podID="4ab49081-96ed-405c-80d9-846b102649d7" containerID="db43910dc7ea528993d2ade2b37707d28815cefe512a5083c0ce66c4ecc9e70d" exitCode=0 Oct 31 14:02:04.478718 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.478660 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" Oct 31 14:02:04.478718 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.478685 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" event={"ID":"4ab49081-96ed-405c-80d9-846b102649d7","Type":"ContainerDied","Data":"db43910dc7ea528993d2ade2b37707d28815cefe512a5083c0ce66c4ecc9e70d"} Oct 31 14:02:04.478829 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.478744 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b" event={"ID":"4ab49081-96ed-405c-80d9-846b102649d7","Type":"ContainerDied","Data":"09643bcdc2a776b9ac0f8cadd581d864e47bdcab9063b90b032cafb9d5b9ff4d"} Oct 31 14:02:04.478829 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.478767 5532 scope.go:122] "RemoveContainer" containerID="db43910dc7ea528993d2ade2b37707d28815cefe512a5083c0ce66c4ecc9e70d" Oct 31 14:02:04.511014 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.510991 5532 scope.go:122] "RemoveContainer" containerID="db43910dc7ea528993d2ade2b37707d28815cefe512a5083c0ce66c4ecc9e70d" Oct 31 14:02:04.511311 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:04.511291 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db43910dc7ea528993d2ade2b37707d28815cefe512a5083c0ce66c4ecc9e70d\": container with ID starting with db43910dc7ea528993d2ade2b37707d28815cefe512a5083c0ce66c4ecc9e70d not found: ID does not exist" containerID="db43910dc7ea528993d2ade2b37707d28815cefe512a5083c0ce66c4ecc9e70d" Oct 31 14:02:04.511370 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.511322 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db43910dc7ea528993d2ade2b37707d28815cefe512a5083c0ce66c4ecc9e70d"} err="failed to get container status \"db43910dc7ea528993d2ade2b37707d28815cefe512a5083c0ce66c4ecc9e70d\": rpc error: code = NotFound desc = could not find container \"db43910dc7ea528993d2ade2b37707d28815cefe512a5083c0ce66c4ecc9e70d\": container with ID starting with db43910dc7ea528993d2ade2b37707d28815cefe512a5083c0ce66c4ecc9e70d not found: ID does not exist" Oct 31 14:02:04.525272 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.525248 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b"] Oct 31 14:02:04.532876 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.532856 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-d85d82-554d685c5-t5x6b"] Oct 31 14:02:04.574793 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.574753 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8qbr\" (UniqueName: \"kubernetes.io/projected/4ab49081-96ed-405c-80d9-846b102649d7-kube-api-access-r8qbr\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:04.574793 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.574787 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:04.574793 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.574802 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ab49081-96ed-405c-80d9-846b102649d7-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:04.588802 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.588772 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx"] Oct 31 14:02:04.594138 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.594111 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-d85d82-r7lzx"] Oct 31 14:02:04.668314 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:04.668273 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9"] Oct 31 14:02:06.003959 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.003925 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ab49081-96ed-405c-80d9-846b102649d7" path="/var/lib/kubelet/pods/4ab49081-96ed-405c-80d9-846b102649d7/volumes" Oct 31 14:02:06.004608 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.004571 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df332b53-ff14-4e58-b911-dd1659c28d87" path="/var/lib/kubelet/pods/df332b53-ff14-4e58-b911-dd1659c28d87/volumes" Oct 31 14:02:06.189752 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:06.189715 5532 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd8e2bd_93c5_4108_a69a_1648d856fede.slice/crio-conmon-77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd8e2bd_93c5_4108_a69a_1648d856fede.slice/crio-77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b.scope\": RecentStats: unable to find data in memory cache]" Oct 31 14:02:06.468635 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.468610 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" Oct 31 14:02:06.500089 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.500058 5532 generic.go:356] "Generic (PLEG): container finished" podID="6cd8e2bd-93c5-4108-a69a-1648d856fede" containerID="77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b" exitCode=0 Oct 31 14:02:06.500253 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.500099 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" Oct 31 14:02:06.500253 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.500141 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" event={"ID":"6cd8e2bd-93c5-4108-a69a-1648d856fede","Type":"ContainerDied","Data":"77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b"} Oct 31 14:02:06.500253 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.500224 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs" event={"ID":"6cd8e2bd-93c5-4108-a69a-1648d856fede","Type":"ContainerDied","Data":"c82c0ee168f498c1b4d8cde17b8785cb8fe14faefda507f4deedee09b8605668"} Oct 31 14:02:06.500253 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.500250 5532 scope.go:122] "RemoveContainer" containerID="77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b" Oct 31 14:02:06.511748 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.511730 5532 scope.go:122] "RemoveContainer" containerID="77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b" Oct 31 14:02:06.512032 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:06.512012 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b\": container with ID starting with 77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b not found: ID does not exist" containerID="77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b" Oct 31 14:02:06.512124 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.512042 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b"} err="failed to get container status \"77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b\": rpc error: code = NotFound desc = could not find container \"77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b\": container with ID starting with 77bc74982c899468c55c2141d5d7995c5120a593f1a73f67efbb9663bbfb627b not found: ID does not exist" Oct 31 14:02:06.609210 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.609133 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd8e2bd-93c5-4108-a69a-1648d856fede-kube-api-access-4765q\" (UniqueName: \"kubernetes.io/projected/6cd8e2bd-93c5-4108-a69a-1648d856fede-kube-api-access-4765q\") pod \"6cd8e2bd-93c5-4108-a69a-1648d856fede\" (UID: \"6cd8e2bd-93c5-4108-a69a-1648d856fede\") " Oct 31 14:02:06.609210 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.609167 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd8e2bd-93c5-4108-a69a-1648d856fede-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cd8e2bd-93c5-4108-a69a-1648d856fede-plugins-config-volume\") pod \"6cd8e2bd-93c5-4108-a69a-1648d856fede\" (UID: \"6cd8e2bd-93c5-4108-a69a-1648d856fede\") " Oct 31 14:02:06.609583 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.609554 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd8e2bd-93c5-4108-a69a-1648d856fede-plugins-config-volume" pod "6cd8e2bd-93c5-4108-a69a-1648d856fede" (UID: "6cd8e2bd-93c5-4108-a69a-1648d856fede"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:06.609773 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.609758 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6cd8e2bd-93c5-4108-a69a-1648d856fede-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:06.611288 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.611262 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd8e2bd-93c5-4108-a69a-1648d856fede-kube-api-access-4765q" pod "6cd8e2bd-93c5-4108-a69a-1648d856fede" (UID: "6cd8e2bd-93c5-4108-a69a-1648d856fede"). InnerVolumeSpecName "kube-api-access-4765q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:06.711238 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.711198 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4765q\" (UniqueName: \"kubernetes.io/projected/6cd8e2bd-93c5-4108-a69a-1648d856fede-kube-api-access-4765q\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:06.826646 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.826609 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs"] Oct 31 14:02:06.832919 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:06.832892 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-d85d82-655545f7d7-q6jbs"] Oct 31 14:02:08.003424 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:08.003389 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd8e2bd-93c5-4108-a69a-1648d856fede" path="/var/lib/kubelet/pods/6cd8e2bd-93c5-4108-a69a-1648d856fede/volumes" Oct 31 14:02:08.488018 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:08.487984 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh"] Oct 31 14:02:08.488274 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:08.488245 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" containerID="cri-o://f48790afd708ac55b0e686cdfe985aa82b9085d3fe050c7ca9d1392a7627e942" gracePeriod=30 Oct 31 14:02:08.491455 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:08.491430 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx"] Oct 31 14:02:08.491662 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:08.491640 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" podUID="5e752232-e0de-4540-b22d-cf1928b53e31" containerName="istio-proxy" containerID="cri-o://18900f8fe6db6eddc14133d74c5871d51ff5c4f5cd675a7b617a2a47c5cc0d3f" gracePeriod=30 Oct 31 14:02:08.906735 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:08.906698 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" podUID="b45f1106-92ed-4bad-b7c0-aa3a0442793b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:02:09.139639 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.139577 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt"] Oct 31 14:02:09.140154 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.140140 5532 cpu_manager.go:414] "containerMap: removing container" podUID="6cd8e2bd-93c5-4108-a69a-1648d856fede" containerName="payload-processing" Oct 31 14:02:09.140154 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.140154 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd8e2bd-93c5-4108-a69a-1648d856fede" containerName="payload-processing" Oct 31 14:02:09.140240 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.140166 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4ab49081-96ed-405c-80d9-846b102649d7" containerName="maas-api" Oct 31 14:02:09.140240 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.140172 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ab49081-96ed-405c-80d9-846b102649d7" containerName="maas-api" Oct 31 14:02:09.140240 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.140192 5532 cpu_manager.go:414] "containerMap: removing container" podUID="de6b3aca-3802-48c4-9315-2492f01492a0" containerName="payload-processing" Oct 31 14:02:09.140240 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.140197 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de6b3aca-3802-48c4-9315-2492f01492a0" containerName="payload-processing" Oct 31 14:02:09.140365 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.140303 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="de6b3aca-3802-48c4-9315-2492f01492a0" containerName="payload-processing" Oct 31 14:02:09.140365 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.140317 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f63ce148-84ae-4429-9661-77b0e688205a" containerName="istio-proxy" Oct 31 14:02:09.140365 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.140329 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd8e2bd-93c5-4108-a69a-1648d856fede" containerName="payload-processing" Oct 31 14:02:09.140365 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.140336 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ab49081-96ed-405c-80d9-846b102649d7" containerName="maas-api" Oct 31 14:02:09.145248 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.145223 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" Oct 31 14:02:09.158416 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.158350 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt"] Oct 31 14:02:09.182141 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.182114 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:02:09.182376 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.182361 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:02:09.205642 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.205609 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4"] Oct 31 14:02:09.205845 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.205824 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" podUID="bfcecb52-1419-43f3-a98d-a3b6f5a624c4" containerName="istio-proxy" containerID="cri-o://b8abea8d1e3745a8a9086925568a3e64decf8f308cb2bf5e0960945fc582d68b" gracePeriod=30 Oct 31 14:02:09.247255 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.247216 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt\" (UID: \"cb05e1c4-5c79-4743-811a-270d948777e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" Oct 31 14:02:09.247423 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.247325 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt\" (UID: \"cb05e1c4-5c79-4743-811a-270d948777e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" Oct 31 14:02:09.247499 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.247480 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkq64\" (UniqueName: \"kubernetes.io/projected/cb05e1c4-5c79-4743-811a-270d948777e0-kube-api-access-gkq64\") pod \"maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt\" (UID: \"cb05e1c4-5c79-4743-811a-270d948777e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" Oct 31 14:02:09.358397 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.351418 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt\" (UID: \"cb05e1c4-5c79-4743-811a-270d948777e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" Oct 31 14:02:09.358397 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.351498 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt\" (UID: \"cb05e1c4-5c79-4743-811a-270d948777e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" Oct 31 14:02:09.358397 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.351630 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkq64\" (UniqueName: \"kubernetes.io/projected/cb05e1c4-5c79-4743-811a-270d948777e0-kube-api-access-gkq64\") pod \"maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt\" (UID: \"cb05e1c4-5c79-4743-811a-270d948777e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" Oct 31 14:02:09.358397 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.352876 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt\" (UID: \"cb05e1c4-5c79-4743-811a-270d948777e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" Oct 31 14:02:09.366296 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.366210 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt\" (UID: \"cb05e1c4-5c79-4743-811a-270d948777e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" Oct 31 14:02:09.366502 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.366332 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkq64\" (UniqueName: \"kubernetes.io/projected/cb05e1c4-5c79-4743-811a-270d948777e0-kube-api-access-gkq64\") pod \"maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt\" (UID: \"cb05e1c4-5c79-4743-811a-270d948777e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" Oct 31 14:02:09.460437 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.460351 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" Oct 31 14:02:09.668652 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.668587 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" podUID="a5ae713b-d2f7-45c0-bf5c-166d580bc32d" containerName="payload-pre-processing" containerID="cri-o://f3c0116373c509c978317c2fd17a17233185cd8f4391ba0030df70e8368bbc25" gracePeriod=25 Oct 31 14:02:09.743199 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:09.743169 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt"] Oct 31 14:02:09.744188 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:02:09.744163 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb05e1c4_5c79_4743_811a_270d948777e0.slice/crio-715e051ed2b13f522ff0469814f1e5578ffe9465b968d8482df184c22363b1fc WatchSource:0}: Error finding container 715e051ed2b13f522ff0469814f1e5578ffe9465b968d8482df184c22363b1fc: Status 404 returned error can't find the container with id 715e051ed2b13f522ff0469814f1e5578ffe9465b968d8482df184c22363b1fc Oct 31 14:02:10.399560 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:10.399526 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:02:10.399967 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:10.399621 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" podUID="5e752232-e0de-4540-b22d-cf1928b53e31" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:02:10.554916 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:10.554875 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" event={"ID":"cb05e1c4-5c79-4743-811a-270d948777e0","Type":"ContainerStarted","Data":"dca19a4423cc4dd258cccd32bbd588c5b152623a7ef4389b248953d5919280ab"} Oct 31 14:02:10.555092 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:10.554940 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" event={"ID":"cb05e1c4-5c79-4743-811a-270d948777e0","Type":"ContainerStarted","Data":"715e051ed2b13f522ff0469814f1e5578ffe9465b968d8482df184c22363b1fc"} Oct 31 14:02:10.590181 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:10.590131 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" podStartSLOduration=1.590114013 podStartE2EDuration="1.590114013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:10.588296325 +0000 UTC m=+3498.734040167" watchObservedRunningTime="2026-10-31 14:02:10.590114013 +0000 UTC m=+3498.735857842" Oct 31 14:02:10.862747 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:10.862708 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr"] Oct 31 14:02:10.863005 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:10.862980 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" podUID="4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0" containerName="maas-api" containerID="cri-o://5808c72f31cb540c90cd52cbb911753da1115e288c01e4ed1fbfa26c636fe585" gracePeriod=30 Oct 31 14:02:11.042062 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.042018 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd"] Oct 31 14:02:11.047821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.046769 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5e85a70-zqknd"] Oct 31 14:02:11.173617 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.168562 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 14:02:11.177614 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.177278 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-maas-api-tls\") pod \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " Oct 31 14:02:11.177614 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.177348 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-kube-api-access-gjhsg\" (UniqueName: \"kubernetes.io/projected/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-kube-api-access-gjhsg\") pod \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " Oct 31 14:02:11.177614 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.177382 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-metrics-certs\") pod \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " Oct 31 14:02:11.177876 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.177619 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-openshift-service-ca\") pod \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\" (UID: \"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0\") " Oct 31 14:02:11.179848 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.178463 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-openshift-service-ca" pod "4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0" (UID: "4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:11.186474 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.186425 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-kube-api-access-gjhsg" pod "4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0" (UID: "4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0"). InnerVolumeSpecName "kube-api-access-gjhsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:11.187882 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.187846 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-maas-api-tls" pod "4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0" (UID: "4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:11.195844 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.195809 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-metrics-certs" pod "4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0" (UID: "4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:11.279429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.279348 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:11.279429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.279379 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjhsg\" (UniqueName: \"kubernetes.io/projected/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-kube-api-access-gjhsg\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:11.279429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.279389 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:11.279429 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.279399 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:11.564410 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.564312 5532 generic.go:356] "Generic (PLEG): container finished" podID="cb05e1c4-5c79-4743-811a-270d948777e0" containerID="dca19a4423cc4dd258cccd32bbd588c5b152623a7ef4389b248953d5919280ab" exitCode=0 Oct 31 14:02:11.564875 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.564406 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" event={"ID":"cb05e1c4-5c79-4743-811a-270d948777e0","Type":"ContainerDied","Data":"dca19a4423cc4dd258cccd32bbd588c5b152623a7ef4389b248953d5919280ab"} Oct 31 14:02:11.566339 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.566314 5532 generic.go:356] "Generic (PLEG): container finished" podID="4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0" containerID="5808c72f31cb540c90cd52cbb911753da1115e288c01e4ed1fbfa26c636fe585" exitCode=0 Oct 31 14:02:11.566517 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.566354 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" Oct 31 14:02:11.566517 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.566352 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" event={"ID":"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0","Type":"ContainerDied","Data":"5808c72f31cb540c90cd52cbb911753da1115e288c01e4ed1fbfa26c636fe585"} Oct 31 14:02:11.566517 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.566493 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr" event={"ID":"4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0","Type":"ContainerDied","Data":"f21ce1afe80cb44ee6f8e7180f7d2c7b8b19ef57efc9ca5982a1f78cdbf400a8"} Oct 31 14:02:11.566643 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.566523 5532 scope.go:122] "RemoveContainer" containerID="5808c72f31cb540c90cd52cbb911753da1115e288c01e4ed1fbfa26c636fe585" Oct 31 14:02:11.577407 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.577388 5532 scope.go:122] "RemoveContainer" containerID="5808c72f31cb540c90cd52cbb911753da1115e288c01e4ed1fbfa26c636fe585" Oct 31 14:02:11.577680 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:11.577660 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5808c72f31cb540c90cd52cbb911753da1115e288c01e4ed1fbfa26c636fe585\": container with ID starting with 5808c72f31cb540c90cd52cbb911753da1115e288c01e4ed1fbfa26c636fe585 not found: ID does not exist" containerID="5808c72f31cb540c90cd52cbb911753da1115e288c01e4ed1fbfa26c636fe585" Oct 31 14:02:11.577773 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.577690 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5808c72f31cb540c90cd52cbb911753da1115e288c01e4ed1fbfa26c636fe585"} err="failed to get container status \"5808c72f31cb540c90cd52cbb911753da1115e288c01e4ed1fbfa26c636fe585\": rpc error: code = NotFound desc = could not find container \"5808c72f31cb540c90cd52cbb911753da1115e288c01e4ed1fbfa26c636fe585\": container with ID starting with 5808c72f31cb540c90cd52cbb911753da1115e288c01e4ed1fbfa26c636fe585 not found: ID does not exist" Oct 31 14:02:11.601535 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.601508 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr"] Oct 31 14:02:11.606632 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:11.606611 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-abdcb1-7db4d74585-7gfqr"] Oct 31 14:02:12.003783 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.003735 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0" path="/var/lib/kubelet/pods/4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0/volumes" Oct 31 14:02:12.004152 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.004138 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ae5236-c47d-4182-9d85-9b8131f11783" path="/var/lib/kubelet/pods/56ae5236-c47d-4182-9d85-9b8131f11783/volumes" Oct 31 14:02:12.401988 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.401952 5532 patch_prober.go:29] interesting pod/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:02:12.402165 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.402029 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:02:12.727458 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.727423 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" Oct 31 14:02:12.799785 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.799744 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb05e1c4-5c79-4743-811a-270d948777e0-kube-api-access-gkq64\" (UniqueName: \"kubernetes.io/projected/cb05e1c4-5c79-4743-811a-270d948777e0-kube-api-access-gkq64\") pod \"cb05e1c4-5c79-4743-811a-270d948777e0\" (UID: \"cb05e1c4-5c79-4743-811a-270d948777e0\") " Oct 31 14:02:12.799959 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.799802 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-openshift-ca\") pod \"cb05e1c4-5c79-4743-811a-270d948777e0\" (UID: \"cb05e1c4-5c79-4743-811a-270d948777e0\") " Oct 31 14:02:12.799959 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.799842 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-xks-ca\") pod \"cb05e1c4-5c79-4743-811a-270d948777e0\" (UID: \"cb05e1c4-5c79-4743-811a-270d948777e0\") " Oct 31 14:02:12.800155 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.800133 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-openshift-ca" pod "cb05e1c4-5c79-4743-811a-270d948777e0" (UID: "cb05e1c4-5c79-4743-811a-270d948777e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:12.800412 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.800392 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:12.801990 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.801965 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb05e1c4-5c79-4743-811a-270d948777e0-kube-api-access-gkq64" pod "cb05e1c4-5c79-4743-811a-270d948777e0" (UID: "cb05e1c4-5c79-4743-811a-270d948777e0"). InnerVolumeSpecName "kube-api-access-gkq64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:12.801990 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.801967 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-xks-ca" pod "cb05e1c4-5c79-4743-811a-270d948777e0" (UID: "cb05e1c4-5c79-4743-811a-270d948777e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:12.901846 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.901810 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkq64\" (UniqueName: \"kubernetes.io/projected/cb05e1c4-5c79-4743-811a-270d948777e0-kube-api-access-gkq64\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:12.901846 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:12.901841 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb05e1c4-5c79-4743-811a-270d948777e0-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.584944 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.584916 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" Oct 31 14:02:13.585087 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.584950 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt" event={"ID":"cb05e1c4-5c79-4743-811a-270d948777e0","Type":"ContainerDied","Data":"715e051ed2b13f522ff0469814f1e5578ffe9465b968d8482df184c22363b1fc"} Oct 31 14:02:13.585087 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.585030 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="715e051ed2b13f522ff0469814f1e5578ffe9465b968d8482df184c22363b1fc" Oct 31 14:02:13.780316 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.780022 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 14:02:13.822546 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.822519 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 14:02:13.823975 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.823945 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-envoy\") pod \"5e752232-e0de-4540-b22d-cf1928b53e31\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " Oct 31 14:02:13.824142 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.824001 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e752232-e0de-4540-b22d-cf1928b53e31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e752232-e0de-4540-b22d-cf1928b53e31-istiod-ca-cert\") pod \"5e752232-e0de-4540-b22d-cf1928b53e31\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " Oct 31 14:02:13.824142 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.824063 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-kube-api-access-s85s4\" (UniqueName: \"kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-kube-api-access-s85s4\") pod \"5e752232-e0de-4540-b22d-cf1928b53e31\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " Oct 31 14:02:13.824142 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.824096 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-data\") pod \"5e752232-e0de-4540-b22d-cf1928b53e31\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " Oct 31 14:02:13.824142 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.824134 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-socket\") pod \"5e752232-e0de-4540-b22d-cf1928b53e31\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " Oct 31 14:02:13.824395 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.824200 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e752232-e0de-4540-b22d-cf1928b53e31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e752232-e0de-4540-b22d-cf1928b53e31-istio-podinfo\") pod \"5e752232-e0de-4540-b22d-cf1928b53e31\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " Oct 31 14:02:13.824395 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.824252 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-istio-token\" (UniqueName: \"kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-istio-token\") pod \"5e752232-e0de-4540-b22d-cf1928b53e31\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " Oct 31 14:02:13.824395 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.824306 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-credential-socket\") pod \"5e752232-e0de-4540-b22d-cf1928b53e31\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " Oct 31 14:02:13.824617 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.824475 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-certs\") pod \"5e752232-e0de-4540-b22d-cf1928b53e31\" (UID: \"5e752232-e0de-4540-b22d-cf1928b53e31\") " Oct 31 14:02:13.824853 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.824825 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e752232-e0de-4540-b22d-cf1928b53e31-istiod-ca-cert" pod "5e752232-e0de-4540-b22d-cf1928b53e31" (UID: "5e752232-e0de-4540-b22d-cf1928b53e31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:13.825087 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.825065 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-certs" pod "5e752232-e0de-4540-b22d-cf1928b53e31" (UID: "5e752232-e0de-4540-b22d-cf1928b53e31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:13.825261 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.825236 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e752232-e0de-4540-b22d-cf1928b53e31-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.825345 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.825267 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.825449 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.825394 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-socket" pod "5e752232-e0de-4540-b22d-cf1928b53e31" (UID: "5e752232-e0de-4540-b22d-cf1928b53e31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:13.826824 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.825513 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-credential-socket" pod "5e752232-e0de-4540-b22d-cf1928b53e31" (UID: "5e752232-e0de-4540-b22d-cf1928b53e31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:13.826824 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.825565 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-data" pod "5e752232-e0de-4540-b22d-cf1928b53e31" (UID: "5e752232-e0de-4540-b22d-cf1928b53e31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:13.829562 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.829536 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-envoy" pod "5e752232-e0de-4540-b22d-cf1928b53e31" (UID: "5e752232-e0de-4540-b22d-cf1928b53e31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:13.830232 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.830206 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-istio-token" pod "5e752232-e0de-4540-b22d-cf1928b53e31" (UID: "5e752232-e0de-4540-b22d-cf1928b53e31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:13.830232 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.830220 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e752232-e0de-4540-b22d-cf1928b53e31-istio-podinfo" pod "5e752232-e0de-4540-b22d-cf1928b53e31" (UID: "5e752232-e0de-4540-b22d-cf1928b53e31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:02:13.830563 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.830536 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-kube-api-access-s85s4" pod "5e752232-e0de-4540-b22d-cf1928b53e31" (UID: "5e752232-e0de-4540-b22d-cf1928b53e31"). InnerVolumeSpecName "kube-api-access-s85s4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:13.926543 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.926512 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-envoy\") pod \"1242bc05-fa46-4097-afaf-36a1e19194a9\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " Oct 31 14:02:13.926879 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.926579 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1242bc05-fa46-4097-afaf-36a1e19194a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1242bc05-fa46-4097-afaf-36a1e19194a9-istiod-ca-cert\") pod \"1242bc05-fa46-4097-afaf-36a1e19194a9\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " Oct 31 14:02:13.926879 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.926639 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-socket\") pod \"1242bc05-fa46-4097-afaf-36a1e19194a9\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " Oct 31 14:02:13.926879 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.926683 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-certs\") pod \"1242bc05-fa46-4097-afaf-36a1e19194a9\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " Oct 31 14:02:13.926879 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.926709 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-data\") pod \"1242bc05-fa46-4097-afaf-36a1e19194a9\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " Oct 31 14:02:13.926879 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.926785 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-podinfo\") pod \"1242bc05-fa46-4097-afaf-36a1e19194a9\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " Oct 31 14:02:13.926879 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.926812 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-token\" (UniqueName: \"kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-token\") pod \"1242bc05-fa46-4097-afaf-36a1e19194a9\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " Oct 31 14:02:13.927262 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.926894 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-credential-socket\") pod \"1242bc05-fa46-4097-afaf-36a1e19194a9\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " Oct 31 14:02:13.927262 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.926938 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-socket" pod "1242bc05-fa46-4097-afaf-36a1e19194a9" (UID: "1242bc05-fa46-4097-afaf-36a1e19194a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:13.927262 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927026 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-certs" pod "1242bc05-fa46-4097-afaf-36a1e19194a9" (UID: "1242bc05-fa46-4097-afaf-36a1e19194a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:13.927262 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927124 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-credential-socket" pod "1242bc05-fa46-4097-afaf-36a1e19194a9" (UID: "1242bc05-fa46-4097-afaf-36a1e19194a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:13.927262 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927131 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-kube-api-access-tv798\" (UniqueName: \"kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-kube-api-access-tv798\") pod \"1242bc05-fa46-4097-afaf-36a1e19194a9\" (UID: \"1242bc05-fa46-4097-afaf-36a1e19194a9\") " Oct 31 14:02:13.927262 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927183 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1242bc05-fa46-4097-afaf-36a1e19194a9-istiod-ca-cert" pod "1242bc05-fa46-4097-afaf-36a1e19194a9" (UID: "1242bc05-fa46-4097-afaf-36a1e19194a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:13.927530 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927074 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-data" pod "1242bc05-fa46-4097-afaf-36a1e19194a9" (UID: "1242bc05-fa46-4097-afaf-36a1e19194a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:13.927798 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927774 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s85s4\" (UniqueName: \"kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-kube-api-access-s85s4\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.927869 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927807 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.927869 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927822 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.927869 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927835 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.927869 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927848 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.927869 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927865 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e752232-e0de-4540-b22d-cf1928b53e31-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.928040 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927878 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.928040 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927892 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e752232-e0de-4540-b22d-cf1928b53e31-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.928040 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927905 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.928040 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927919 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e752232-e0de-4540-b22d-cf1928b53e31-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.928040 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927931 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1242bc05-fa46-4097-afaf-36a1e19194a9-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.928040 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.927946 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:13.929025 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.928978 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-envoy" pod "1242bc05-fa46-4097-afaf-36a1e19194a9" (UID: "1242bc05-fa46-4097-afaf-36a1e19194a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:13.929132 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.929052 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-podinfo" pod "1242bc05-fa46-4097-afaf-36a1e19194a9" (UID: "1242bc05-fa46-4097-afaf-36a1e19194a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:02:13.929690 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.929401 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-kube-api-access-tv798" pod "1242bc05-fa46-4097-afaf-36a1e19194a9" (UID: "1242bc05-fa46-4097-afaf-36a1e19194a9"). InnerVolumeSpecName "kube-api-access-tv798". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:13.929690 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:13.929436 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-token" pod "1242bc05-fa46-4097-afaf-36a1e19194a9" (UID: "1242bc05-fa46-4097-afaf-36a1e19194a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:14.029259 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.029232 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:14.029259 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.029256 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:14.029418 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.029266 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv798\" (UniqueName: \"kubernetes.io/projected/1242bc05-fa46-4097-afaf-36a1e19194a9-kube-api-access-tv798\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:14.029418 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.029276 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1242bc05-fa46-4097-afaf-36a1e19194a9-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:14.058899 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.058872 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz"] Oct 31 14:02:14.070398 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.070269 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs"] Oct 31 14:02:14.473933 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.473905 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 14:02:14.536288 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.536257 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-podinfo\") pod \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " Oct 31 14:02:14.536481 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.536333 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-certs\") pod \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " Oct 31 14:02:14.536481 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.536473 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-data\") pod \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " Oct 31 14:02:14.536632 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.536546 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-socket\") pod \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " Oct 31 14:02:14.536632 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.536613 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istiod-ca-cert\") pod \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " Oct 31 14:02:14.536755 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.536644 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-certs" pod "bfcecb52-1419-43f3-a98d-a3b6f5a624c4" (UID: "bfcecb52-1419-43f3-a98d-a3b6f5a624c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:14.536755 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.536659 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-kube-api-access-6ldr2\" (UniqueName: \"kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-kube-api-access-6ldr2\") pod \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " Oct 31 14:02:14.536755 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.536736 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-credential-socket\") pod \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " Oct 31 14:02:14.536995 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.536766 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-data" pod "bfcecb52-1419-43f3-a98d-a3b6f5a624c4" (UID: "bfcecb52-1419-43f3-a98d-a3b6f5a624c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:14.536995 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.536773 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-token\" (UniqueName: \"kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-token\") pod \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " Oct 31 14:02:14.536995 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.536799 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-socket" pod "bfcecb52-1419-43f3-a98d-a3b6f5a624c4" (UID: "bfcecb52-1419-43f3-a98d-a3b6f5a624c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:14.536995 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.536834 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-envoy\") pod \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\" (UID: \"bfcecb52-1419-43f3-a98d-a3b6f5a624c4\") " Oct 31 14:02:14.537237 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.537007 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istiod-ca-cert" pod "bfcecb52-1419-43f3-a98d-a3b6f5a624c4" (UID: "bfcecb52-1419-43f3-a98d-a3b6f5a624c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:14.537299 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.537253 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-credential-socket" pod "bfcecb52-1419-43f3-a98d-a3b6f5a624c4" (UID: "bfcecb52-1419-43f3-a98d-a3b6f5a624c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:14.537417 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.537400 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:14.537482 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.537424 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:14.537482 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.537440 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:14.537482 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.537454 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:14.537482 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.537468 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:14.539273 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.539243 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-podinfo" pod "bfcecb52-1419-43f3-a98d-a3b6f5a624c4" (UID: "bfcecb52-1419-43f3-a98d-a3b6f5a624c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:02:14.539558 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.539528 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-envoy" pod "bfcecb52-1419-43f3-a98d-a3b6f5a624c4" (UID: "bfcecb52-1419-43f3-a98d-a3b6f5a624c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:14.539725 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.539624 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-kube-api-access-6ldr2" pod "bfcecb52-1419-43f3-a98d-a3b6f5a624c4" (UID: "bfcecb52-1419-43f3-a98d-a3b6f5a624c4"). InnerVolumeSpecName "kube-api-access-6ldr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:14.539877 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.539853 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-token" pod "bfcecb52-1419-43f3-a98d-a3b6f5a624c4" (UID: "bfcecb52-1419-43f3-a98d-a3b6f5a624c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:14.593871 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.593795 5532 generic.go:356] "Generic (PLEG): container finished" podID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerID="f48790afd708ac55b0e686cdfe985aa82b9085d3fe050c7ca9d1392a7627e942" exitCode=0 Oct 31 14:02:14.593871 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.593846 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" Oct 31 14:02:14.594060 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.593877 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" event={"ID":"1242bc05-fa46-4097-afaf-36a1e19194a9","Type":"ContainerDied","Data":"f48790afd708ac55b0e686cdfe985aa82b9085d3fe050c7ca9d1392a7627e942"} Oct 31 14:02:14.594060 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.593932 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh" event={"ID":"1242bc05-fa46-4097-afaf-36a1e19194a9","Type":"ContainerDied","Data":"afdf345218603bc956a245f38a1955892ce1e174fd50985d2235d21e78402999"} Oct 31 14:02:14.594060 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.593964 5532 scope.go:122] "RemoveContainer" containerID="f48790afd708ac55b0e686cdfe985aa82b9085d3fe050c7ca9d1392a7627e942" Oct 31 14:02:14.595552 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.595526 5532 generic.go:356] "Generic (PLEG): container finished" podID="5e752232-e0de-4540-b22d-cf1928b53e31" containerID="18900f8fe6db6eddc14133d74c5871d51ff5c4f5cd675a7b617a2a47c5cc0d3f" exitCode=0 Oct 31 14:02:14.595707 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.595580 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" Oct 31 14:02:14.595707 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.595623 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" event={"ID":"5e752232-e0de-4540-b22d-cf1928b53e31","Type":"ContainerDied","Data":"18900f8fe6db6eddc14133d74c5871d51ff5c4f5cd675a7b617a2a47c5cc0d3f"} Oct 31 14:02:14.595707 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.595671 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx" event={"ID":"5e752232-e0de-4540-b22d-cf1928b53e31","Type":"ContainerDied","Data":"66b8ff9e376733db93baceed6f142109a8424cf661f98a5309fd0cbb22bee6aa"} Oct 31 14:02:14.597412 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.597389 5532 generic.go:356] "Generic (PLEG): container finished" podID="bfcecb52-1419-43f3-a98d-a3b6f5a624c4" containerID="b8abea8d1e3745a8a9086925568a3e64decf8f308cb2bf5e0960945fc582d68b" exitCode=0 Oct 31 14:02:14.597557 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.597429 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" Oct 31 14:02:14.597557 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.597486 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" event={"ID":"bfcecb52-1419-43f3-a98d-a3b6f5a624c4","Type":"ContainerDied","Data":"b8abea8d1e3745a8a9086925568a3e64decf8f308cb2bf5e0960945fc582d68b"} Oct 31 14:02:14.597557 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.597528 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4" event={"ID":"bfcecb52-1419-43f3-a98d-a3b6f5a624c4","Type":"ContainerDied","Data":"b383b979024be0632524c0e92c9e8140866b27629394fc0ece7f737ed6ec906f"} Oct 31 14:02:14.605793 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.605582 5532 scope.go:122] "RemoveContainer" containerID="229d94709835a6fe0570be3d7d56ce5f087c0f3fee03cbd2879862be3baa12b1" Oct 31 14:02:14.616636 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.616617 5532 scope.go:122] "RemoveContainer" containerID="f48790afd708ac55b0e686cdfe985aa82b9085d3fe050c7ca9d1392a7627e942" Oct 31 14:02:14.616893 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:14.616874 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48790afd708ac55b0e686cdfe985aa82b9085d3fe050c7ca9d1392a7627e942\": container with ID starting with f48790afd708ac55b0e686cdfe985aa82b9085d3fe050c7ca9d1392a7627e942 not found: ID does not exist" containerID="f48790afd708ac55b0e686cdfe985aa82b9085d3fe050c7ca9d1392a7627e942" Oct 31 14:02:14.616969 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.616908 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48790afd708ac55b0e686cdfe985aa82b9085d3fe050c7ca9d1392a7627e942"} err="failed to get container status \"f48790afd708ac55b0e686cdfe985aa82b9085d3fe050c7ca9d1392a7627e942\": rpc error: code = NotFound desc = could not find container \"f48790afd708ac55b0e686cdfe985aa82b9085d3fe050c7ca9d1392a7627e942\": container with ID starting with f48790afd708ac55b0e686cdfe985aa82b9085d3fe050c7ca9d1392a7627e942 not found: ID does not exist" Oct 31 14:02:14.616969 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.616932 5532 scope.go:122] "RemoveContainer" containerID="229d94709835a6fe0570be3d7d56ce5f087c0f3fee03cbd2879862be3baa12b1" Oct 31 14:02:14.617152 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:14.617134 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229d94709835a6fe0570be3d7d56ce5f087c0f3fee03cbd2879862be3baa12b1\": container with ID starting with 229d94709835a6fe0570be3d7d56ce5f087c0f3fee03cbd2879862be3baa12b1 not found: ID does not exist" containerID="229d94709835a6fe0570be3d7d56ce5f087c0f3fee03cbd2879862be3baa12b1" Oct 31 14:02:14.617198 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.617160 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229d94709835a6fe0570be3d7d56ce5f087c0f3fee03cbd2879862be3baa12b1"} err="failed to get container status \"229d94709835a6fe0570be3d7d56ce5f087c0f3fee03cbd2879862be3baa12b1\": rpc error: code = NotFound desc = could not find container \"229d94709835a6fe0570be3d7d56ce5f087c0f3fee03cbd2879862be3baa12b1\": container with ID starting with 229d94709835a6fe0570be3d7d56ce5f087c0f3fee03cbd2879862be3baa12b1 not found: ID does not exist" Oct 31 14:02:14.617198 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.617176 5532 scope.go:122] "RemoveContainer" containerID="18900f8fe6db6eddc14133d74c5871d51ff5c4f5cd675a7b617a2a47c5cc0d3f" Oct 31 14:02:14.626549 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.626533 5532 scope.go:122] "RemoveContainer" containerID="18900f8fe6db6eddc14133d74c5871d51ff5c4f5cd675a7b617a2a47c5cc0d3f" Oct 31 14:02:14.626785 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:14.626766 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18900f8fe6db6eddc14133d74c5871d51ff5c4f5cd675a7b617a2a47c5cc0d3f\": container with ID starting with 18900f8fe6db6eddc14133d74c5871d51ff5c4f5cd675a7b617a2a47c5cc0d3f not found: ID does not exist" containerID="18900f8fe6db6eddc14133d74c5871d51ff5c4f5cd675a7b617a2a47c5cc0d3f" Oct 31 14:02:14.626845 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.626792 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18900f8fe6db6eddc14133d74c5871d51ff5c4f5cd675a7b617a2a47c5cc0d3f"} err="failed to get container status \"18900f8fe6db6eddc14133d74c5871d51ff5c4f5cd675a7b617a2a47c5cc0d3f\": rpc error: code = NotFound desc = could not find container \"18900f8fe6db6eddc14133d74c5871d51ff5c4f5cd675a7b617a2a47c5cc0d3f\": container with ID starting with 18900f8fe6db6eddc14133d74c5871d51ff5c4f5cd675a7b617a2a47c5cc0d3f not found: ID does not exist" Oct 31 14:02:14.626845 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.626811 5532 scope.go:122] "RemoveContainer" containerID="b8abea8d1e3745a8a9086925568a3e64decf8f308cb2bf5e0960945fc582d68b" Oct 31 14:02:14.636381 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.636360 5532 scope.go:122] "RemoveContainer" containerID="b8abea8d1e3745a8a9086925568a3e64decf8f308cb2bf5e0960945fc582d68b" Oct 31 14:02:14.636448 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.636394 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx"] Oct 31 14:02:14.636684 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:14.636655 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8abea8d1e3745a8a9086925568a3e64decf8f308cb2bf5e0960945fc582d68b\": container with ID starting with b8abea8d1e3745a8a9086925568a3e64decf8f308cb2bf5e0960945fc582d68b not found: ID does not exist" containerID="b8abea8d1e3745a8a9086925568a3e64decf8f308cb2bf5e0960945fc582d68b" Oct 31 14:02:14.636813 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.636690 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8abea8d1e3745a8a9086925568a3e64decf8f308cb2bf5e0960945fc582d68b"} err="failed to get container status \"b8abea8d1e3745a8a9086925568a3e64decf8f308cb2bf5e0960945fc582d68b\": rpc error: code = NotFound desc = could not find container \"b8abea8d1e3745a8a9086925568a3e64decf8f308cb2bf5e0960945fc582d68b\": container with ID starting with b8abea8d1e3745a8a9086925568a3e64decf8f308cb2bf5e0960945fc582d68b not found: ID does not exist" Oct 31 14:02:14.638308 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.638288 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:14.638381 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.638310 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:14.638381 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.638320 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:14.638381 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.638331 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6ldr2\" (UniqueName: \"kubernetes.io/projected/bfcecb52-1419-43f3-a98d-a3b6f5a624c4-kube-api-access-6ldr2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:14.647008 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.646980 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-cr8gx"] Oct 31 14:02:14.666265 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.666230 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh"] Oct 31 14:02:14.670905 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.670884 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-14ba5a-openshift-default-66ccc567f-pppvh"] Oct 31 14:02:14.686037 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.685892 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4"] Oct 31 14:02:14.689106 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.689083 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-15dc2f-openshift-default-b5c8767cc-2gpc4"] Oct 31 14:02:14.909633 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.909588 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n"] Oct 31 14:02:14.910381 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910364 5532 cpu_manager.go:414] "containerMap: removing container" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" Oct 31 14:02:14.910434 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910382 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" Oct 31 14:02:14.910434 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910409 5532 cpu_manager.go:414] "containerMap: removing container" podUID="5e752232-e0de-4540-b22d-cf1928b53e31" containerName="istio-proxy" Oct 31 14:02:14.910434 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910426 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e752232-e0de-4540-b22d-cf1928b53e31" containerName="istio-proxy" Oct 31 14:02:14.910547 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910443 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0" containerName="maas-api" Oct 31 14:02:14.910547 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910451 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0" containerName="maas-api" Oct 31 14:02:14.910547 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910462 5532 cpu_manager.go:414] "containerMap: removing container" podUID="bfcecb52-1419-43f3-a98d-a3b6f5a624c4" containerName="istio-proxy" Oct 31 14:02:14.910547 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910469 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfcecb52-1419-43f3-a98d-a3b6f5a624c4" containerName="istio-proxy" Oct 31 14:02:14.910547 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910493 5532 cpu_manager.go:414] "containerMap: removing container" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" Oct 31 14:02:14.910547 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910501 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" Oct 31 14:02:14.910547 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910526 5532 cpu_manager.go:414] "containerMap: removing container" podUID="cb05e1c4-5c79-4743-811a-270d948777e0" containerName="revoke-keys" Oct 31 14:02:14.910547 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910533 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb05e1c4-5c79-4743-811a-270d948777e0" containerName="revoke-keys" Oct 31 14:02:14.910828 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910655 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb05e1c4-5c79-4743-811a-270d948777e0" containerName="revoke-keys" Oct 31 14:02:14.910828 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910675 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" Oct 31 14:02:14.910828 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910686 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e752232-e0de-4540-b22d-cf1928b53e31" containerName="istio-proxy" Oct 31 14:02:14.910828 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910697 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4151ebe4-8e7c-439b-a2cb-8d3b4343d6f0" containerName="maas-api" Oct 31 14:02:14.910828 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910707 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfcecb52-1419-43f3-a98d-a3b6f5a624c4" containerName="istio-proxy" Oct 31 14:02:14.910828 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.910719 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" containerName="istio-proxy" Oct 31 14:02:14.923680 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.917665 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:14.924907 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.924885 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-1b75f9-serving-cert\"" Oct 31 14:02:14.927466 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:14.927440 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n"] Oct 31 14:02:15.051461 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.051425 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:15.051676 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.051636 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-metrics-certs\") pod \"maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:15.051762 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.051698 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d55c6c-187f-46d2-8994-cb0ef0412ca9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:15.051866 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.051847 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5lfv\" (UniqueName: \"kubernetes.io/projected/17d55c6c-187f-46d2-8994-cb0ef0412ca9-kube-api-access-f5lfv\") pod \"maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:15.154541 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.154488 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5lfv\" (UniqueName: \"kubernetes.io/projected/17d55c6c-187f-46d2-8994-cb0ef0412ca9-kube-api-access-f5lfv\") pod \"maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:15.155112 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.155077 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:15.155254 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.155214 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-metrics-certs\") pod \"maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:15.155328 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.155262 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d55c6c-187f-46d2-8994-cb0ef0412ca9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:15.156100 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.156073 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d55c6c-187f-46d2-8994-cb0ef0412ca9-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:15.158231 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.158203 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-maas-api-tls\") pod \"maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:15.158423 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.158403 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-metrics-certs\") pod \"maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:15.170070 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.170003 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5lfv\" (UniqueName: \"kubernetes.io/projected/17d55c6c-187f-46d2-8994-cb0ef0412ca9-kube-api-access-f5lfv\") pod \"maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:15.238342 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.238308 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:15.528748 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.528717 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n"] Oct 31 14:02:15.529708 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:02:15.529678 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17d55c6c_187f_46d2_8994_cb0ef0412ca9.slice/crio-af7dd1c6a1ddd942c533720af43bafb618b09da4ea766236173544c93e0dc5df WatchSource:0}: Error finding container af7dd1c6a1ddd942c533720af43bafb618b09da4ea766236173544c93e0dc5df: Status 404 returned error can't find the container with id af7dd1c6a1ddd942c533720af43bafb618b09da4ea766236173544c93e0dc5df Oct 31 14:02:15.531176 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.531156 5532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 14:02:15.606191 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:15.606148 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" event={"ID":"17d55c6c-187f-46d2-8994-cb0ef0412ca9","Type":"ContainerStarted","Data":"af7dd1c6a1ddd942c533720af43bafb618b09da4ea766236173544c93e0dc5df"} Oct 31 14:02:16.003187 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:16.003159 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1242bc05-fa46-4097-afaf-36a1e19194a9" path="/var/lib/kubelet/pods/1242bc05-fa46-4097-afaf-36a1e19194a9/volumes" Oct 31 14:02:16.003731 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:16.003715 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e752232-e0de-4540-b22d-cf1928b53e31" path="/var/lib/kubelet/pods/5e752232-e0de-4540-b22d-cf1928b53e31/volumes" Oct 31 14:02:16.004156 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:16.004143 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfcecb52-1419-43f3-a98d-a3b6f5a624c4" path="/var/lib/kubelet/pods/bfcecb52-1419-43f3-a98d-a3b6f5a624c4/volumes" Oct 31 14:02:16.618156 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:16.618121 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" event={"ID":"17d55c6c-187f-46d2-8994-cb0ef0412ca9","Type":"ContainerStarted","Data":"8fa460a4db75bc540b780e43dec01bc13909f28bca203fad0db86775edceed4e"} Oct 31 14:02:16.618361 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:16.618185 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:16.640717 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:16.640667 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" podStartSLOduration=2.238674467 podStartE2EDuration="2.640649064s" totalImagesPullingTime="401.974597ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:16.639280959 +0000 UTC m=+3504.785024801" watchObservedRunningTime="2026-10-31 14:02:16.640649064 +0000 UTC m=+3504.786392893" Oct 31 14:02:17.248098 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:17.248063 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" podUID="a5ae713b-d2f7-45c0-bf5c-166d580bc32d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:02:18.267226 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.267199 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" Oct 31 14:02:18.396963 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.396935 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45f1106-92ed-4bad-b7c0-aa3a0442793b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45f1106-92ed-4bad-b7c0-aa3a0442793b-plugins-config-volume\") pod \"b45f1106-92ed-4bad-b7c0-aa3a0442793b\" (UID: \"b45f1106-92ed-4bad-b7c0-aa3a0442793b\") " Oct 31 14:02:18.397158 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.396997 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45f1106-92ed-4bad-b7c0-aa3a0442793b-kube-api-access-tg2w6\" (UniqueName: \"kubernetes.io/projected/b45f1106-92ed-4bad-b7c0-aa3a0442793b-kube-api-access-tg2w6\") pod \"b45f1106-92ed-4bad-b7c0-aa3a0442793b\" (UID: \"b45f1106-92ed-4bad-b7c0-aa3a0442793b\") " Oct 31 14:02:18.397377 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.397351 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45f1106-92ed-4bad-b7c0-aa3a0442793b-plugins-config-volume" pod "b45f1106-92ed-4bad-b7c0-aa3a0442793b" (UID: "b45f1106-92ed-4bad-b7c0-aa3a0442793b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:18.399175 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.399147 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45f1106-92ed-4bad-b7c0-aa3a0442793b-kube-api-access-tg2w6" pod "b45f1106-92ed-4bad-b7c0-aa3a0442793b" (UID: "b45f1106-92ed-4bad-b7c0-aa3a0442793b"). InnerVolumeSpecName "kube-api-access-tg2w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:18.498397 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.498317 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg2w6\" (UniqueName: \"kubernetes.io/projected/b45f1106-92ed-4bad-b7c0-aa3a0442793b-kube-api-access-tg2w6\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:18.498397 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.498346 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45f1106-92ed-4bad-b7c0-aa3a0442793b-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:18.634399 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.634365 5532 generic.go:356] "Generic (PLEG): container finished" podID="b45f1106-92ed-4bad-b7c0-aa3a0442793b" containerID="011936b68cb4fbbd96119431b3062007e1fda2275ecf43592ca1381f6c33f015" exitCode=0 Oct 31 14:02:18.634650 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.634431 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" Oct 31 14:02:18.634650 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.634451 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" event={"ID":"b45f1106-92ed-4bad-b7c0-aa3a0442793b","Type":"ContainerDied","Data":"011936b68cb4fbbd96119431b3062007e1fda2275ecf43592ca1381f6c33f015"} Oct 31 14:02:18.634650 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.634549 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr" event={"ID":"b45f1106-92ed-4bad-b7c0-aa3a0442793b","Type":"ContainerDied","Data":"6c398d001f3eeca308e58e0d56af841998d14c920558e1c75ddcc1edf9a447ab"} Oct 31 14:02:18.634650 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.634581 5532 scope.go:122] "RemoveContainer" containerID="011936b68cb4fbbd96119431b3062007e1fda2275ecf43592ca1381f6c33f015" Oct 31 14:02:18.649921 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.649891 5532 scope.go:122] "RemoveContainer" containerID="011936b68cb4fbbd96119431b3062007e1fda2275ecf43592ca1381f6c33f015" Oct 31 14:02:18.650739 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:18.650706 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"011936b68cb4fbbd96119431b3062007e1fda2275ecf43592ca1381f6c33f015\": container with ID starting with 011936b68cb4fbbd96119431b3062007e1fda2275ecf43592ca1381f6c33f015 not found: ID does not exist" containerID="011936b68cb4fbbd96119431b3062007e1fda2275ecf43592ca1381f6c33f015" Oct 31 14:02:18.650869 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.650747 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"011936b68cb4fbbd96119431b3062007e1fda2275ecf43592ca1381f6c33f015"} err="failed to get container status \"011936b68cb4fbbd96119431b3062007e1fda2275ecf43592ca1381f6c33f015\": rpc error: code = NotFound desc = could not find container \"011936b68cb4fbbd96119431b3062007e1fda2275ecf43592ca1381f6c33f015\": container with ID starting with 011936b68cb4fbbd96119431b3062007e1fda2275ecf43592ca1381f6c33f015 not found: ID does not exist" Oct 31 14:02:18.662607 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.662571 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr"] Oct 31 14:02:18.665219 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:18.665196 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-15dc2f-5c94c8878c-24mmr"] Oct 31 14:02:19.253589 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:19.253551 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q"] Oct 31 14:02:20.004972 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:20.004940 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45f1106-92ed-4bad-b7c0-aa3a0442793b" path="/var/lib/kubelet/pods/b45f1106-92ed-4bad-b7c0-aa3a0442793b/volumes" Oct 31 14:02:20.746796 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:20.746758 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6"] Oct 31 14:02:20.747059 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:20.747020 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" podUID="32f40b74-b0b4-4f94-a08e-3bd06c339993" containerName="istio-proxy" containerID="cri-o://29213cd50f64676d77addf2ee1e589b1106e6d1846f5c85a285be0d6bbd6b2bd" gracePeriod=30 Oct 31 14:02:21.743484 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.743453 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5"] Oct 31 14:02:21.744006 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.743991 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b45f1106-92ed-4bad-b7c0-aa3a0442793b" containerName="payload-processing" Oct 31 14:02:21.744006 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.744006 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45f1106-92ed-4bad-b7c0-aa3a0442793b" containerName="payload-processing" Oct 31 14:02:21.744138 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.744128 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45f1106-92ed-4bad-b7c0-aa3a0442793b" containerName="payload-processing" Oct 31 14:02:21.749102 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.749083 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" Oct 31 14:02:21.755999 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.755974 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:02:21.756535 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.756512 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5"] Oct 31 14:02:21.757423 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.757402 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:02:21.840447 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.840408 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6652ad-qj9f5\" (UID: \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" Oct 31 14:02:21.840668 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.840478 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d59p2\" (UniqueName: \"kubernetes.io/projected/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-kube-api-access-d59p2\") pod \"maas-api-revoke-sub-7e6652ad-qj9f5\" (UID: \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" Oct 31 14:02:21.840778 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.840754 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6652ad-qj9f5\" (UID: \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" Oct 31 14:02:21.942178 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.942142 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6652ad-qj9f5\" (UID: \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" Oct 31 14:02:21.942394 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.942224 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d59p2\" (UniqueName: \"kubernetes.io/projected/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-kube-api-access-d59p2\") pod \"maas-api-revoke-sub-7e6652ad-qj9f5\" (UID: \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" Oct 31 14:02:21.942476 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.942407 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6652ad-qj9f5\" (UID: \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" Oct 31 14:02:21.943051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.943030 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e6652ad-qj9f5\" (UID: \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" Oct 31 14:02:21.944617 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.944586 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e6652ad-qj9f5\" (UID: \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" Oct 31 14:02:21.955853 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:21.955831 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d59p2\" (UniqueName: \"kubernetes.io/projected/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-kube-api-access-d59p2\") pod \"maas-api-revoke-sub-7e6652ad-qj9f5\" (UID: \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" Oct 31 14:02:22.027845 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:22.027744 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f"] Oct 31 14:02:22.030861 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:22.030834 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c68ba1d5-fc58f"] Oct 31 14:02:22.065561 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:22.065530 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" Oct 31 14:02:22.352470 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:22.352437 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5"] Oct 31 14:02:22.354139 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:02:22.354113 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded22fd13_4e70_4c82_9b23_be39ca96e9e5.slice/crio-0a2858d0772c0afeba09ffc405f87fece57620684157c9b173a1889ea6c25207 WatchSource:0}: Error finding container 0a2858d0772c0afeba09ffc405f87fece57620684157c9b173a1889ea6c25207: Status 404 returned error can't find the container with id 0a2858d0772c0afeba09ffc405f87fece57620684157c9b173a1889ea6c25207 Oct 31 14:02:22.631982 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:22.631910 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:02:22.670006 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:22.669975 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" event={"ID":"ed22fd13-4e70-4c82-9b23-be39ca96e9e5","Type":"ContainerStarted","Data":"5103df28eb2bb9f461842120748baaff2767a3f550bd46ac2f1f46244a87fdf5"} Oct 31 14:02:22.670170 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:22.670031 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" event={"ID":"ed22fd13-4e70-4c82-9b23-be39ca96e9e5","Type":"ContainerStarted","Data":"0a2858d0772c0afeba09ffc405f87fece57620684157c9b173a1889ea6c25207"} Oct 31 14:02:22.694640 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:22.694570 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" podStartSLOduration=1.69455439 podStartE2EDuration="1.69455439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:22.692291552 +0000 UTC m=+3510.838035383" watchObservedRunningTime="2026-10-31 14:02:22.69455439 +0000 UTC m=+3510.840298232" Oct 31 14:02:23.679093 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:23.679050 5532 generic.go:356] "Generic (PLEG): container finished" podID="ed22fd13-4e70-4c82-9b23-be39ca96e9e5" containerID="5103df28eb2bb9f461842120748baaff2767a3f550bd46ac2f1f46244a87fdf5" exitCode=0 Oct 31 14:02:23.679462 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:23.679140 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" event={"ID":"ed22fd13-4e70-4c82-9b23-be39ca96e9e5","Type":"ContainerDied","Data":"5103df28eb2bb9f461842120748baaff2767a3f550bd46ac2f1f46244a87fdf5"} Oct 31 14:02:24.003449 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:24.003370 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c7b96f-35de-4027-b62d-8bfff3670de5" path="/var/lib/kubelet/pods/60c7b96f-35de-4027-b62d-8bfff3670de5/volumes" Oct 31 14:02:24.254655 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:24.254527 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" podUID="27513f81-9d1f-407b-bca8-86c3d9659bf0" containerName="payload-processing" containerID="cri-o://f79f349a648ce097da929ab5d5934688ada1e93099741d5d68cfebe59baea9af" gracePeriod=25 Oct 31 14:02:24.858918 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:24.858896 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" Oct 31 14:02:24.983333 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:24.983264 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-kube-api-access-d59p2\" (UniqueName: \"kubernetes.io/projected/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-kube-api-access-d59p2\") pod \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\" (UID: \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\") " Oct 31 14:02:24.983477 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:24.983436 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-xks-ca\") pod \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\" (UID: \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\") " Oct 31 14:02:24.983531 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:24.983482 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-openshift-ca\") pod \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\" (UID: \"ed22fd13-4e70-4c82-9b23-be39ca96e9e5\") " Oct 31 14:02:24.983953 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:24.983922 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-openshift-ca" pod "ed22fd13-4e70-4c82-9b23-be39ca96e9e5" (UID: "ed22fd13-4e70-4c82-9b23-be39ca96e9e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:24.985430 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:24.985408 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-xks-ca" pod "ed22fd13-4e70-4c82-9b23-be39ca96e9e5" (UID: "ed22fd13-4e70-4c82-9b23-be39ca96e9e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:24.985520 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:24.985443 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-kube-api-access-d59p2" pod "ed22fd13-4e70-4c82-9b23-be39ca96e9e5" (UID: "ed22fd13-4e70-4c82-9b23-be39ca96e9e5"). InnerVolumeSpecName "kube-api-access-d59p2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:25.086367 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:25.086328 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:25.086367 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:25.086366 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d59p2\" (UniqueName: \"kubernetes.io/projected/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-kube-api-access-d59p2\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:25.086832 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:25.086380 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed22fd13-4e70-4c82-9b23-be39ca96e9e5-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:25.695138 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:25.695106 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" Oct 31 14:02:25.695330 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:25.695135 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5" event={"ID":"ed22fd13-4e70-4c82-9b23-be39ca96e9e5","Type":"ContainerDied","Data":"0a2858d0772c0afeba09ffc405f87fece57620684157c9b173a1889ea6c25207"} Oct 31 14:02:25.695330 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:25.695209 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a2858d0772c0afeba09ffc405f87fece57620684157c9b173a1889ea6c25207" Oct 31 14:02:26.020301 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.020274 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 14:02:26.095985 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.095955 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32f40b74-b0b4-4f94-a08e-3bd06c339993-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f40b74-b0b4-4f94-a08e-3bd06c339993-istiod-ca-cert\") pod \"32f40b74-b0b4-4f94-a08e-3bd06c339993\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " Oct 31 14:02:26.096170 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.096002 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-data\") pod \"32f40b74-b0b4-4f94-a08e-3bd06c339993\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " Oct 31 14:02:26.096170 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.096080 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-token\" (UniqueName: \"kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-token\") pod \"32f40b74-b0b4-4f94-a08e-3bd06c339993\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " Oct 31 14:02:26.096170 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.096124 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-envoy\") pod \"32f40b74-b0b4-4f94-a08e-3bd06c339993\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " Oct 31 14:02:26.096339 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.096174 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-socket\") pod \"32f40b74-b0b4-4f94-a08e-3bd06c339993\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " Oct 31 14:02:26.096339 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.096237 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-certs\") pod \"32f40b74-b0b4-4f94-a08e-3bd06c339993\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " Oct 31 14:02:26.096454 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.096341 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-podinfo\") pod \"32f40b74-b0b4-4f94-a08e-3bd06c339993\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " Oct 31 14:02:26.096454 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.096358 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-data" pod "32f40b74-b0b4-4f94-a08e-3bd06c339993" (UID: "32f40b74-b0b4-4f94-a08e-3bd06c339993"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:26.096454 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.096367 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32f40b74-b0b4-4f94-a08e-3bd06c339993-istiod-ca-cert" pod "32f40b74-b0b4-4f94-a08e-3bd06c339993" (UID: "32f40b74-b0b4-4f94-a08e-3bd06c339993"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:26.096454 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.096401 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-kube-api-access-cpfnz\" (UniqueName: \"kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-kube-api-access-cpfnz\") pod \"32f40b74-b0b4-4f94-a08e-3bd06c339993\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " Oct 31 14:02:26.096454 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.096436 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-credential-socket\") pod \"32f40b74-b0b4-4f94-a08e-3bd06c339993\" (UID: \"32f40b74-b0b4-4f94-a08e-3bd06c339993\") " Oct 31 14:02:26.096758 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.096617 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-certs" pod "32f40b74-b0b4-4f94-a08e-3bd06c339993" (UID: "32f40b74-b0b4-4f94-a08e-3bd06c339993"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:26.096758 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.096643 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-socket" pod "32f40b74-b0b4-4f94-a08e-3bd06c339993" (UID: "32f40b74-b0b4-4f94-a08e-3bd06c339993"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:26.096874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.096839 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-credential-socket" pod "32f40b74-b0b4-4f94-a08e-3bd06c339993" (UID: "32f40b74-b0b4-4f94-a08e-3bd06c339993"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:26.097067 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.097040 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:26.097177 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.097072 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32f40b74-b0b4-4f94-a08e-3bd06c339993-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:26.097177 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.097089 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:26.097177 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.097104 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:26.097177 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.097118 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:26.098541 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.098513 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-envoy" pod "32f40b74-b0b4-4f94-a08e-3bd06c339993" (UID: "32f40b74-b0b4-4f94-a08e-3bd06c339993"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:26.098696 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.098680 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-kube-api-access-cpfnz" pod "32f40b74-b0b4-4f94-a08e-3bd06c339993" (UID: "32f40b74-b0b4-4f94-a08e-3bd06c339993"). InnerVolumeSpecName "kube-api-access-cpfnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:26.098853 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.098832 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-podinfo" pod "32f40b74-b0b4-4f94-a08e-3bd06c339993" (UID: "32f40b74-b0b4-4f94-a08e-3bd06c339993"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:02:26.098947 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.098855 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-token" pod "32f40b74-b0b4-4f94-a08e-3bd06c339993" (UID: "32f40b74-b0b4-4f94-a08e-3bd06c339993"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:26.179892 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.179855 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj"] Oct 31 14:02:26.184757 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.184729 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww"] Oct 31 14:02:26.198771 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.198740 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:26.198904 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.198775 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpfnz\" (UniqueName: \"kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-kube-api-access-cpfnz\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:26.198904 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.198792 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:26.198904 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.198806 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32f40b74-b0b4-4f94-a08e-3bd06c339993-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:26.209571 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.209546 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2"] Oct 31 14:02:26.703809 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.703772 5532 generic.go:356] "Generic (PLEG): container finished" podID="32f40b74-b0b4-4f94-a08e-3bd06c339993" containerID="29213cd50f64676d77addf2ee1e589b1106e6d1846f5c85a285be0d6bbd6b2bd" exitCode=0 Oct 31 14:02:26.703985 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.703842 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" Oct 31 14:02:26.703985 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.703863 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" event={"ID":"32f40b74-b0b4-4f94-a08e-3bd06c339993","Type":"ContainerDied","Data":"29213cd50f64676d77addf2ee1e589b1106e6d1846f5c85a285be0d6bbd6b2bd"} Oct 31 14:02:26.703985 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.703920 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6" event={"ID":"32f40b74-b0b4-4f94-a08e-3bd06c339993","Type":"ContainerDied","Data":"b0c59aa7e0a29b426dd91f5700f8e569bbe8de89fc1bc867452435520d99f78e"} Oct 31 14:02:26.703985 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.703951 5532 scope.go:122] "RemoveContainer" containerID="29213cd50f64676d77addf2ee1e589b1106e6d1846f5c85a285be0d6bbd6b2bd" Oct 31 14:02:26.717422 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.717400 5532 scope.go:122] "RemoveContainer" containerID="29213cd50f64676d77addf2ee1e589b1106e6d1846f5c85a285be0d6bbd6b2bd" Oct 31 14:02:26.717748 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:26.717724 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29213cd50f64676d77addf2ee1e589b1106e6d1846f5c85a285be0d6bbd6b2bd\": container with ID starting with 29213cd50f64676d77addf2ee1e589b1106e6d1846f5c85a285be0d6bbd6b2bd not found: ID does not exist" containerID="29213cd50f64676d77addf2ee1e589b1106e6d1846f5c85a285be0d6bbd6b2bd" Oct 31 14:02:26.717822 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.717756 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29213cd50f64676d77addf2ee1e589b1106e6d1846f5c85a285be0d6bbd6b2bd"} err="failed to get container status \"29213cd50f64676d77addf2ee1e589b1106e6d1846f5c85a285be0d6bbd6b2bd\": rpc error: code = NotFound desc = could not find container \"29213cd50f64676d77addf2ee1e589b1106e6d1846f5c85a285be0d6bbd6b2bd\": container with ID starting with 29213cd50f64676d77addf2ee1e589b1106e6d1846f5c85a285be0d6bbd6b2bd not found: ID does not exist" Oct 31 14:02:26.732308 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.732287 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6"] Oct 31 14:02:26.734837 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:26.734817 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-abdcb1-openshift-default-6b5875986f-74ns6"] Oct 31 14:02:27.247880 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:27.247843 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" podUID="a5ae713b-d2f7-45c0-bf5c-166d580bc32d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:02:28.003552 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:28.003514 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32f40b74-b0b4-4f94-a08e-3bd06c339993" path="/var/lib/kubelet/pods/32f40b74-b0b4-4f94-a08e-3bd06c339993/volumes" Oct 31 14:02:29.081691 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.081651 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" containerID="cri-o://204fcbc9dbddb55792c9be72e3d4cdce02c7c801fa647b7db1e68737de0507ef" gracePeriod=45 Oct 31 14:02:29.091033 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.091003 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" containerID="cri-o://afe9827608151bfdc447788864c79df94f847df42c96447683f23b31095b76a8" gracePeriod=45 Oct 31 14:02:29.366370 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.366342 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 14:02:29.393295 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.393264 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 14:02:29.536909 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.536873 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tls-certs\" (UniqueName: \"kubernetes.io/secret/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tls-certs\") pod \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " Oct 31 14:02:29.537091 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.536942 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-model-cache\") pod \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " Oct 31 14:02:29.537091 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.536995 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tls-certs\") pod \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " Oct 31 14:02:29.537091 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537016 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-home\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-home\") pod \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " Oct 31 14:02:29.537091 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537036 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-kube-api-access-ctsm4\" (UniqueName: \"kubernetes.io/projected/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-kube-api-access-ctsm4\") pod \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " Oct 31 14:02:29.537091 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537064 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tmp-dir\") pod \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " Oct 31 14:02:29.537393 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537095 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-kube-api-access-jgzc9\" (UniqueName: \"kubernetes.io/projected/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-kube-api-access-jgzc9\") pod \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " Oct 31 14:02:29.537393 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537158 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-home\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-home\") pod \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " Oct 31 14:02:29.537393 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537223 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-model-cache" pod "c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" (UID: "c0a427ed-a515-41f3-9d4a-7d2e03d4efb8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:29.537393 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537239 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tmp-dir\") pod \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " Oct 31 14:02:29.537393 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537325 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tmp-dir" pod "c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" (UID: "c0a427ed-a515-41f3-9d4a-7d2e03d4efb8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:29.537393 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537337 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-dshm\") pod \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\" (UID: \"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8\") " Oct 31 14:02:29.537810 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537398 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-model-cache\") pod \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " Oct 31 14:02:29.537810 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537340 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-home" pod "c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" (UID: "c0a427ed-a515-41f3-9d4a-7d2e03d4efb8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:29.537810 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537420 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tmp-dir" pod "849b0d63-9287-4d49-b4c6-dbdb6af0f37a" (UID: "849b0d63-9287-4d49-b4c6-dbdb6af0f37a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:29.537810 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537464 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-dshm\") pod \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\" (UID: \"849b0d63-9287-4d49-b4c6-dbdb6af0f37a\") " Oct 31 14:02:29.537810 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537554 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-home" pod "849b0d63-9287-4d49-b4c6-dbdb6af0f37a" (UID: "849b0d63-9287-4d49-b4c6-dbdb6af0f37a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:29.537810 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.537777 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-model-cache" pod "849b0d63-9287-4d49-b4c6-dbdb6af0f37a" (UID: "849b0d63-9287-4d49-b4c6-dbdb6af0f37a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:29.538156 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.538111 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:29.538156 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.538132 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:29.538156 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.538147 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:29.538391 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.538160 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:29.538391 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.538173 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:29.538391 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.538185 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:29.539537 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.539508 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tls-certs" pod "849b0d63-9287-4d49-b4c6-dbdb6af0f37a" (UID: "849b0d63-9287-4d49-b4c6-dbdb6af0f37a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:29.539676 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.539505 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tls-certs" pod "c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" (UID: "c0a427ed-a515-41f3-9d4a-7d2e03d4efb8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:29.539796 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.539777 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-dshm" pod "c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" (UID: "c0a427ed-a515-41f3-9d4a-7d2e03d4efb8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:29.539847 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.539817 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-kube-api-access-ctsm4" pod "c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" (UID: "c0a427ed-a515-41f3-9d4a-7d2e03d4efb8"). InnerVolumeSpecName "kube-api-access-ctsm4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:29.539938 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.539918 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-dshm" pod "849b0d63-9287-4d49-b4c6-dbdb6af0f37a" (UID: "849b0d63-9287-4d49-b4c6-dbdb6af0f37a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:29.540265 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.540251 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-kube-api-access-jgzc9" pod "849b0d63-9287-4d49-b4c6-dbdb6af0f37a" (UID: "849b0d63-9287-4d49-b4c6-dbdb6af0f37a"). InnerVolumeSpecName "kube-api-access-jgzc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:29.639373 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.639286 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:29.639373 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.639315 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:29.639373 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.639324 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:29.639373 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.639333 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctsm4\" (UniqueName: \"kubernetes.io/projected/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-kube-api-access-ctsm4\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:29.639373 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.639343 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgzc9\" (UniqueName: \"kubernetes.io/projected/849b0d63-9287-4d49-b4c6-dbdb6af0f37a-kube-api-access-jgzc9\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:29.639373 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.639352 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:29.727105 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.727072 5532 generic.go:356] "Generic (PLEG): container finished" podID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerID="afe9827608151bfdc447788864c79df94f847df42c96447683f23b31095b76a8" exitCode=0 Oct 31 14:02:29.727255 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.727125 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" Oct 31 14:02:29.727255 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.727155 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" event={"ID":"849b0d63-9287-4d49-b4c6-dbdb6af0f37a","Type":"ContainerDied","Data":"afe9827608151bfdc447788864c79df94f847df42c96447683f23b31095b76a8"} Oct 31 14:02:29.727255 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.727197 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" event={"ID":"849b0d63-9287-4d49-b4c6-dbdb6af0f37a","Type":"ContainerDied","Data":"07a0c28bf498ad2b1c91ca3c3823e456c7135f8786011ec8ceaa48c8e4037199"} Oct 31 14:02:29.727255 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.727215 5532 scope.go:122] "RemoveContainer" containerID="afe9827608151bfdc447788864c79df94f847df42c96447683f23b31095b76a8" Oct 31 14:02:29.728944 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.728918 5532 generic.go:356] "Generic (PLEG): container finished" podID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerID="204fcbc9dbddb55792c9be72e3d4cdce02c7c801fa647b7db1e68737de0507ef" exitCode=0 Oct 31 14:02:29.729071 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.728959 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" Oct 31 14:02:29.729071 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.728986 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" event={"ID":"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8","Type":"ContainerDied","Data":"204fcbc9dbddb55792c9be72e3d4cdce02c7c801fa647b7db1e68737de0507ef"} Oct 31 14:02:29.729071 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.729017 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz" event={"ID":"c0a427ed-a515-41f3-9d4a-7d2e03d4efb8","Type":"ContainerDied","Data":"70ab02fa043a5d0ff5da8a708a9e07195675e7cdda2a00da458b2ee7a8d70e34"} Oct 31 14:02:29.756999 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.756974 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz"] Oct 31 14:02:29.761438 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.761384 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/premium-w5-6784d6-kserve-75dc9b6d46-gcbwz"] Oct 31 14:02:29.776381 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.776360 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs"] Oct 31 14:02:29.785691 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.785297 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs"] Oct 31 14:02:29.801217 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.801190 5532 scope.go:122] "RemoveContainer" containerID="afe9827608151bfdc447788864c79df94f847df42c96447683f23b31095b76a8" Oct 31 14:02:29.801859 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:29.801830 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afe9827608151bfdc447788864c79df94f847df42c96447683f23b31095b76a8\": container with ID starting with afe9827608151bfdc447788864c79df94f847df42c96447683f23b31095b76a8 not found: ID does not exist" containerID="afe9827608151bfdc447788864c79df94f847df42c96447683f23b31095b76a8" Oct 31 14:02:29.801973 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.801868 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afe9827608151bfdc447788864c79df94f847df42c96447683f23b31095b76a8"} err="failed to get container status \"afe9827608151bfdc447788864c79df94f847df42c96447683f23b31095b76a8\": rpc error: code = NotFound desc = could not find container \"afe9827608151bfdc447788864c79df94f847df42c96447683f23b31095b76a8\": container with ID starting with afe9827608151bfdc447788864c79df94f847df42c96447683f23b31095b76a8 not found: ID does not exist" Oct 31 14:02:29.801973 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.801895 5532 scope.go:122] "RemoveContainer" containerID="204fcbc9dbddb55792c9be72e3d4cdce02c7c801fa647b7db1e68737de0507ef" Oct 31 14:02:29.811997 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.811963 5532 scope.go:122] "RemoveContainer" containerID="204fcbc9dbddb55792c9be72e3d4cdce02c7c801fa647b7db1e68737de0507ef" Oct 31 14:02:29.812443 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:29.812397 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"204fcbc9dbddb55792c9be72e3d4cdce02c7c801fa647b7db1e68737de0507ef\": container with ID starting with 204fcbc9dbddb55792c9be72e3d4cdce02c7c801fa647b7db1e68737de0507ef not found: ID does not exist" containerID="204fcbc9dbddb55792c9be72e3d4cdce02c7c801fa647b7db1e68737de0507ef" Oct 31 14:02:29.812553 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.812436 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"204fcbc9dbddb55792c9be72e3d4cdce02c7c801fa647b7db1e68737de0507ef"} err="failed to get container status \"204fcbc9dbddb55792c9be72e3d4cdce02c7c801fa647b7db1e68737de0507ef\": rpc error: code = NotFound desc = could not find container \"204fcbc9dbddb55792c9be72e3d4cdce02c7c801fa647b7db1e68737de0507ef\": container with ID starting with 204fcbc9dbddb55792c9be72e3d4cdce02c7c801fa647b7db1e68737de0507ef not found: ID does not exist" Oct 31 14:02:29.927280 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:29.927255 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" Oct 31 14:02:30.002836 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.002792 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" path="/var/lib/kubelet/pods/849b0d63-9287-4d49-b4c6-dbdb6af0f37a/volumes" Oct 31 14:02:30.003350 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.003328 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" path="/var/lib/kubelet/pods/c0a427ed-a515-41f3-9d4a-7d2e03d4efb8/volumes" Oct 31 14:02:30.043218 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.043182 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-kube-api-access-l2c77\" (UniqueName: \"kubernetes.io/projected/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-kube-api-access-l2c77\") pod \"a5ae713b-d2f7-45c0-bf5c-166d580bc32d\" (UID: \"a5ae713b-d2f7-45c0-bf5c-166d580bc32d\") " Oct 31 14:02:30.043366 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.043279 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-plugins-config-volume\") pod \"a5ae713b-d2f7-45c0-bf5c-166d580bc32d\" (UID: \"a5ae713b-d2f7-45c0-bf5c-166d580bc32d\") " Oct 31 14:02:30.043688 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.043662 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-plugins-config-volume" pod "a5ae713b-d2f7-45c0-bf5c-166d580bc32d" (UID: "a5ae713b-d2f7-45c0-bf5c-166d580bc32d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:30.045304 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.045283 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-kube-api-access-l2c77" pod "a5ae713b-d2f7-45c0-bf5c-166d580bc32d" (UID: "a5ae713b-d2f7-45c0-bf5c-166d580bc32d"). InnerVolumeSpecName "kube-api-access-l2c77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:30.145211 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.145169 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:30.145211 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.145201 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2c77\" (UniqueName: \"kubernetes.io/projected/a5ae713b-d2f7-45c0-bf5c-166d580bc32d-kube-api-access-l2c77\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:30.387715 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.387590 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-6784d6/simulator-w5-6784d6-kserve-5dff7dd579-c9kqs" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.51:8000/ready\": context deadline exceeded" Oct 31 14:02:30.739450 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.739421 5532 generic.go:356] "Generic (PLEG): container finished" podID="a5ae713b-d2f7-45c0-bf5c-166d580bc32d" containerID="f3c0116373c509c978317c2fd17a17233185cd8f4391ba0030df70e8368bbc25" exitCode=0 Oct 31 14:02:30.739623 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.739466 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" Oct 31 14:02:30.739623 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.739500 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" event={"ID":"a5ae713b-d2f7-45c0-bf5c-166d580bc32d","Type":"ContainerDied","Data":"f3c0116373c509c978317c2fd17a17233185cd8f4391ba0030df70e8368bbc25"} Oct 31 14:02:30.739623 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.739555 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9" event={"ID":"a5ae713b-d2f7-45c0-bf5c-166d580bc32d","Type":"ContainerDied","Data":"50b2e4a7dba614e089a57de2da9a06e0065884f029508da63a063f8ab8bfd7e2"} Oct 31 14:02:30.739623 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.739578 5532 scope.go:122] "RemoveContainer" containerID="f3c0116373c509c978317c2fd17a17233185cd8f4391ba0030df70e8368bbc25" Oct 31 14:02:30.749996 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.749981 5532 scope.go:122] "RemoveContainer" containerID="f3c0116373c509c978317c2fd17a17233185cd8f4391ba0030df70e8368bbc25" Oct 31 14:02:30.750266 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:30.750241 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c0116373c509c978317c2fd17a17233185cd8f4391ba0030df70e8368bbc25\": container with ID starting with f3c0116373c509c978317c2fd17a17233185cd8f4391ba0030df70e8368bbc25 not found: ID does not exist" containerID="f3c0116373c509c978317c2fd17a17233185cd8f4391ba0030df70e8368bbc25" Oct 31 14:02:30.750340 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.750278 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c0116373c509c978317c2fd17a17233185cd8f4391ba0030df70e8368bbc25"} err="failed to get container status \"f3c0116373c509c978317c2fd17a17233185cd8f4391ba0030df70e8368bbc25\": rpc error: code = NotFound desc = could not find container \"f3c0116373c509c978317c2fd17a17233185cd8f4391ba0030df70e8368bbc25\": container with ID starting with f3c0116373c509c978317c2fd17a17233185cd8f4391ba0030df70e8368bbc25 not found: ID does not exist" Oct 31 14:02:30.771330 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.767623 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9"] Oct 31 14:02:30.773280 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:30.773261 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-abdcb1-99bb4999f-ftjr9"] Oct 31 14:02:31.250968 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.250935 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz"] Oct 31 14:02:31.251450 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251437 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" Oct 31 14:02:31.251499 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251451 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" Oct 31 14:02:31.251499 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251465 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ed22fd13-4e70-4c82-9b23-be39ca96e9e5" containerName="revoke-keys" Oct 31 14:02:31.251499 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251469 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed22fd13-4e70-4c82-9b23-be39ca96e9e5" containerName="revoke-keys" Oct 31 14:02:31.251499 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251486 5532 cpu_manager.go:414] "containerMap: removing container" podUID="32f40b74-b0b4-4f94-a08e-3bd06c339993" containerName="istio-proxy" Oct 31 14:02:31.251499 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251492 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32f40b74-b0b4-4f94-a08e-3bd06c339993" containerName="istio-proxy" Oct 31 14:02:31.251675 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251506 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a5ae713b-d2f7-45c0-bf5c-166d580bc32d" containerName="payload-pre-processing" Oct 31 14:02:31.251675 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251513 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5ae713b-d2f7-45c0-bf5c-166d580bc32d" containerName="payload-pre-processing" Oct 31 14:02:31.251675 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251539 5532 cpu_manager.go:414] "containerMap: removing container" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" Oct 31 14:02:31.251675 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251545 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" Oct 31 14:02:31.251675 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251639 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="849b0d63-9287-4d49-b4c6-dbdb6af0f37a" containerName="main" Oct 31 14:02:31.251675 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251650 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0a427ed-a515-41f3-9d4a-7d2e03d4efb8" containerName="main" Oct 31 14:02:31.251675 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251655 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed22fd13-4e70-4c82-9b23-be39ca96e9e5" containerName="revoke-keys" Oct 31 14:02:31.251675 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251663 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="32f40b74-b0b4-4f94-a08e-3bd06c339993" containerName="istio-proxy" Oct 31 14:02:31.251675 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.251671 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5ae713b-d2f7-45c0-bf5c-166d580bc32d" containerName="payload-pre-processing" Oct 31 14:02:31.256377 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.256359 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" Oct 31 14:02:31.264867 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.264848 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1b75f9-dockercfg-4tp2n\"" Oct 31 14:02:31.264977 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.264956 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1b75f9\"" Oct 31 14:02:31.267528 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.267508 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz"] Oct 31 14:02:31.358275 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.358227 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036c2383-7dbd-49ef-a30f-df6bf4d61820-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz\" (UID: \"036c2383-7dbd-49ef-a30f-df6bf4d61820\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" Oct 31 14:02:31.358577 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.358556 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqbsf\" (UniqueName: \"kubernetes.io/projected/036c2383-7dbd-49ef-a30f-df6bf4d61820-kube-api-access-sqbsf\") pod \"payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz\" (UID: \"036c2383-7dbd-49ef-a30f-df6bf4d61820\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" Oct 31 14:02:31.459643 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.459514 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqbsf\" (UniqueName: \"kubernetes.io/projected/036c2383-7dbd-49ef-a30f-df6bf4d61820-kube-api-access-sqbsf\") pod \"payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz\" (UID: \"036c2383-7dbd-49ef-a30f-df6bf4d61820\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" Oct 31 14:02:31.459853 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.459649 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036c2383-7dbd-49ef-a30f-df6bf4d61820-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz\" (UID: \"036c2383-7dbd-49ef-a30f-df6bf4d61820\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" Oct 31 14:02:31.460557 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.460533 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036c2383-7dbd-49ef-a30f-df6bf4d61820-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz\" (UID: \"036c2383-7dbd-49ef-a30f-df6bf4d61820\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" Oct 31 14:02:31.482262 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.482225 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqbsf\" (UniqueName: \"kubernetes.io/projected/036c2383-7dbd-49ef-a30f-df6bf4d61820-kube-api-access-sqbsf\") pod \"payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz\" (UID: \"036c2383-7dbd-49ef-a30f-df6bf4d61820\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" Oct 31 14:02:31.581674 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.579815 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" Oct 31 14:02:31.783570 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:31.783533 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:02:31.871058 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:31.871027 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz"] Oct 31 14:02:31.872370 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:02:31.872347 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036c2383_7dbd_49ef_a30f_df6bf4d61820.slice/crio-ca99322734ecebd82a49dc60d7205c732cd41406914a80fab6ed3a07d32c524e WatchSource:0}: Error finding container ca99322734ecebd82a49dc60d7205c732cd41406914a80fab6ed3a07d32c524e: Status 404 returned error can't find the container with id ca99322734ecebd82a49dc60d7205c732cd41406914a80fab6ed3a07d32c524e Oct 31 14:02:32.003386 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:32.003344 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5ae713b-d2f7-45c0-bf5c-166d580bc32d" path="/var/lib/kubelet/pods/a5ae713b-d2f7-45c0-bf5c-166d580bc32d/volumes" Oct 31 14:02:32.782574 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:32.782534 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" podUID="27513f81-9d1f-407b-bca8-86c3d9659bf0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:02:32.789674 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:32.789638 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" event={"ID":"036c2383-7dbd-49ef-a30f-df6bf4d61820","Type":"ContainerStarted","Data":"28e4e3e2df10503a8bfbbc8268c7adcadc4e72c42bd705abdb2d6c1cd18ec633"} Oct 31 14:02:32.789815 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:32.789697 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" event={"ID":"036c2383-7dbd-49ef-a30f-df6bf4d61820","Type":"ContainerStarted","Data":"ca99322734ecebd82a49dc60d7205c732cd41406914a80fab6ed3a07d32c524e"} Oct 31 14:02:32.789815 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:32.789792 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" Oct 31 14:02:32.822034 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:32.821983 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" podStartSLOduration=1.821969666 podStartE2EDuration="1.821969666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:02:32.819468505 +0000 UTC m=+3520.965212334" watchObservedRunningTime="2026-10-31 14:02:32.821969666 +0000 UTC m=+3520.967713494" Oct 31 14:02:35.617755 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:35.617722 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:02:37.168997 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:37.168955 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j"] Oct 31 14:02:37.968607 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:37.968554 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt"] Oct 31 14:02:37.971011 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:37.970987 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-6784d6-x7rnt"] Oct 31 14:02:38.003127 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:38.003094 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb05e1c4-5c79-4743-811a-270d948777e0" path="/var/lib/kubelet/pods/cb05e1c4-5c79-4743-811a-270d948777e0/volumes" Oct 31 14:02:41.202301 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.202251 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" containerID="cri-o://3dbe0f5ca021aa62c51d3ac34c28c4892636d8c2aa386ecef9a8ff9425b70615" gracePeriod=45 Oct 31 14:02:41.205676 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.205644 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" containerID="cri-o://07230cd7162ff00ec76cd832a1af6c9dcf3e7141c037c2303a60b0510752bb3e" gracePeriod=45 Oct 31 14:02:41.229280 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.229231 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" containerID="cri-o://8287b0a564d9f1f2384fc4d86efe6ff6a1e4ca093e0bdd1a2baccfc8fcbd1be0" gracePeriod=45 Oct 31 14:02:41.587423 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.587115 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 14:02:41.595331 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.595307 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 14:02:41.686376 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.686350 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 14:02:41.737694 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.737610 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-home\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-home\") pod \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " Oct 31 14:02:41.737694 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.737659 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-dshm\") pod \"09ab1bcc-1eac-475b-af39-23d43405e233\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " Oct 31 14:02:41.737935 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.737701 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-dshm\") pod \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " Oct 31 14:02:41.737935 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.737738 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ab1bcc-1eac-475b-af39-23d43405e233-kube-api-access-hxqdw\" (UniqueName: \"kubernetes.io/projected/09ab1bcc-1eac-475b-af39-23d43405e233-kube-api-access-hxqdw\") pod \"09ab1bcc-1eac-475b-af39-23d43405e233\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " Oct 31 14:02:41.737935 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.737781 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ab1bcc-1eac-475b-af39-23d43405e233-tls-certs\" (UniqueName: \"kubernetes.io/secret/09ab1bcc-1eac-475b-af39-23d43405e233-tls-certs\") pod \"09ab1bcc-1eac-475b-af39-23d43405e233\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " Oct 31 14:02:41.737935 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.737845 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-home\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-home\") pod \"09ab1bcc-1eac-475b-af39-23d43405e233\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " Oct 31 14:02:41.737935 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.737873 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tmp-dir\") pod \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " Oct 31 14:02:41.738214 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.737926 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-home" pod "77ce8cc3-c32c-4441-be3c-4ca957f5c7be" (UID: "77ce8cc3-c32c-4441-be3c-4ca957f5c7be"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:41.738214 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.737966 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-tmp-dir\") pod \"09ab1bcc-1eac-475b-af39-23d43405e233\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " Oct 31 14:02:41.738214 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.738013 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-model-cache\") pod \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " Oct 31 14:02:41.738214 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.738077 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-model-cache\") pod \"09ab1bcc-1eac-475b-af39-23d43405e233\" (UID: \"09ab1bcc-1eac-475b-af39-23d43405e233\") " Oct 31 14:02:41.738214 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.738106 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-kube-api-access-mbsr5\" (UniqueName: \"kubernetes.io/projected/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-kube-api-access-mbsr5\") pod \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " Oct 31 14:02:41.738214 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.738141 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tls-certs\" (UniqueName: \"kubernetes.io/secret/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tls-certs\") pod \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\" (UID: \"77ce8cc3-c32c-4441-be3c-4ca957f5c7be\") " Oct 31 14:02:41.738517 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.738344 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-tmp-dir" pod "09ab1bcc-1eac-475b-af39-23d43405e233" (UID: "09ab1bcc-1eac-475b-af39-23d43405e233"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:41.738576 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.738528 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-home" pod "09ab1bcc-1eac-475b-af39-23d43405e233" (UID: "09ab1bcc-1eac-475b-af39-23d43405e233"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:41.744714 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.744678 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-model-cache" pod "77ce8cc3-c32c-4441-be3c-4ca957f5c7be" (UID: "77ce8cc3-c32c-4441-be3c-4ca957f5c7be"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:41.769559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.738710 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.769559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.756285 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.769559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.756330 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.769559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.761821 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-model-cache" pod "09ab1bcc-1eac-475b-af39-23d43405e233" (UID: "09ab1bcc-1eac-475b-af39-23d43405e233"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:41.769559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.762728 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tmp-dir" pod "77ce8cc3-c32c-4441-be3c-4ca957f5c7be" (UID: "77ce8cc3-c32c-4441-be3c-4ca957f5c7be"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:41.769559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.764096 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ab1bcc-1eac-475b-af39-23d43405e233-kube-api-access-hxqdw" pod "09ab1bcc-1eac-475b-af39-23d43405e233" (UID: "09ab1bcc-1eac-475b-af39-23d43405e233"). InnerVolumeSpecName "kube-api-access-hxqdw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:41.769559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.768007 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-kube-api-access-mbsr5" pod "77ce8cc3-c32c-4441-be3c-4ca957f5c7be" (UID: "77ce8cc3-c32c-4441-be3c-4ca957f5c7be"). InnerVolumeSpecName "kube-api-access-mbsr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:41.769559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.769240 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-dshm" pod "77ce8cc3-c32c-4441-be3c-4ca957f5c7be" (UID: "77ce8cc3-c32c-4441-be3c-4ca957f5c7be"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:41.769559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.769356 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-dshm" pod "09ab1bcc-1eac-475b-af39-23d43405e233" (UID: "09ab1bcc-1eac-475b-af39-23d43405e233"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:41.769559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.769465 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ab1bcc-1eac-475b-af39-23d43405e233-tls-certs" pod "09ab1bcc-1eac-475b-af39-23d43405e233" (UID: "09ab1bcc-1eac-475b-af39-23d43405e233"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:41.770099 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.769937 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tls-certs" pod "77ce8cc3-c32c-4441-be3c-4ca957f5c7be" (UID: "77ce8cc3-c32c-4441-be3c-4ca957f5c7be"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:41.856999 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.856951 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-model-cache\") pod \"dccc053f-a13f-496e-8c6e-781f41397b58\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " Oct 31 14:02:41.857188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857017 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-home\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-home\") pod \"dccc053f-a13f-496e-8c6e-781f41397b58\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " Oct 31 14:02:41.857188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857103 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-tmp-dir\") pod \"dccc053f-a13f-496e-8c6e-781f41397b58\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " Oct 31 14:02:41.857188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857142 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-dshm\") pod \"dccc053f-a13f-496e-8c6e-781f41397b58\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " Oct 31 14:02:41.857376 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857220 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccc053f-a13f-496e-8c6e-781f41397b58-kube-api-access-tp24g\" (UniqueName: \"kubernetes.io/projected/dccc053f-a13f-496e-8c6e-781f41397b58-kube-api-access-tp24g\") pod \"dccc053f-a13f-496e-8c6e-781f41397b58\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " Oct 31 14:02:41.857376 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857260 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dccc053f-a13f-496e-8c6e-781f41397b58-tls-certs\" (UniqueName: \"kubernetes.io/secret/dccc053f-a13f-496e-8c6e-781f41397b58-tls-certs\") pod \"dccc053f-a13f-496e-8c6e-781f41397b58\" (UID: \"dccc053f-a13f-496e-8c6e-781f41397b58\") " Oct 31 14:02:41.857493 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857359 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-model-cache" pod "dccc053f-a13f-496e-8c6e-781f41397b58" (UID: "dccc053f-a13f-496e-8c6e-781f41397b58"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:41.857493 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857399 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-home" pod "dccc053f-a13f-496e-8c6e-781f41397b58" (UID: "dccc053f-a13f-496e-8c6e-781f41397b58"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:41.857493 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857419 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-tmp-dir" pod "dccc053f-a13f-496e-8c6e-781f41397b58" (UID: "dccc053f-a13f-496e-8c6e-781f41397b58"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:41.857795 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857775 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.857795 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857796 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.857961 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857806 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxqdw\" (UniqueName: \"kubernetes.io/projected/09ab1bcc-1eac-475b-af39-23d43405e233-kube-api-access-hxqdw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.857961 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857818 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09ab1bcc-1eac-475b-af39-23d43405e233-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.857961 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857828 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.857961 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857840 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.857961 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857851 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.857961 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857860 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.857961 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857871 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.857961 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857883 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09ab1bcc-1eac-475b-af39-23d43405e233-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.857961 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857891 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbsr5\" (UniqueName: \"kubernetes.io/projected/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-kube-api-access-mbsr5\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.857961 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.857901 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77ce8cc3-c32c-4441-be3c-4ca957f5c7be-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.859374 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.859348 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-dshm" pod "dccc053f-a13f-496e-8c6e-781f41397b58" (UID: "dccc053f-a13f-496e-8c6e-781f41397b58"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:41.859557 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.859539 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccc053f-a13f-496e-8c6e-781f41397b58-kube-api-access-tp24g" pod "dccc053f-a13f-496e-8c6e-781f41397b58" (UID: "dccc053f-a13f-496e-8c6e-781f41397b58"). InnerVolumeSpecName "kube-api-access-tp24g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:41.859841 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.859825 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dccc053f-a13f-496e-8c6e-781f41397b58-tls-certs" pod "dccc053f-a13f-496e-8c6e-781f41397b58" (UID: "dccc053f-a13f-496e-8c6e-781f41397b58"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:41.881651 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.881617 5532 generic.go:356] "Generic (PLEG): container finished" podID="09ab1bcc-1eac-475b-af39-23d43405e233" containerID="07230cd7162ff00ec76cd832a1af6c9dcf3e7141c037c2303a60b0510752bb3e" exitCode=0 Oct 31 14:02:41.881804 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.881659 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" Oct 31 14:02:41.881804 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.881691 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" event={"ID":"09ab1bcc-1eac-475b-af39-23d43405e233","Type":"ContainerDied","Data":"07230cd7162ff00ec76cd832a1af6c9dcf3e7141c037c2303a60b0510752bb3e"} Oct 31 14:02:41.881804 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.881734 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww" event={"ID":"09ab1bcc-1eac-475b-af39-23d43405e233","Type":"ContainerDied","Data":"5365e510d99fcb9a167862e637721534fb864ca0502e5e62b21561d3585ff5d6"} Oct 31 14:02:41.881804 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.881753 5532 scope.go:122] "RemoveContainer" containerID="07230cd7162ff00ec76cd832a1af6c9dcf3e7141c037c2303a60b0510752bb3e" Oct 31 14:02:41.883329 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.883294 5532 generic.go:356] "Generic (PLEG): container finished" podID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerID="3dbe0f5ca021aa62c51d3ac34c28c4892636d8c2aa386ecef9a8ff9425b70615" exitCode=0 Oct 31 14:02:41.883461 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.883334 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" Oct 31 14:02:41.883461 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.883350 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" event={"ID":"77ce8cc3-c32c-4441-be3c-4ca957f5c7be","Type":"ContainerDied","Data":"3dbe0f5ca021aa62c51d3ac34c28c4892636d8c2aa386ecef9a8ff9425b70615"} Oct 31 14:02:41.883461 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.883396 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj" event={"ID":"77ce8cc3-c32c-4441-be3c-4ca957f5c7be","Type":"ContainerDied","Data":"f672b3c116a3d79806c66bedcf34c7256a53b04f5901881157c87defa6eff285"} Oct 31 14:02:41.885176 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.885149 5532 generic.go:356] "Generic (PLEG): container finished" podID="dccc053f-a13f-496e-8c6e-781f41397b58" containerID="8287b0a564d9f1f2384fc4d86efe6ff6a1e4ca093e0bdd1a2baccfc8fcbd1be0" exitCode=0 Oct 31 14:02:41.885332 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.885196 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" Oct 31 14:02:41.885332 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.885233 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" event={"ID":"dccc053f-a13f-496e-8c6e-781f41397b58","Type":"ContainerDied","Data":"8287b0a564d9f1f2384fc4d86efe6ff6a1e4ca093e0bdd1a2baccfc8fcbd1be0"} Oct 31 14:02:41.885332 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.885267 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2" event={"ID":"dccc053f-a13f-496e-8c6e-781f41397b58","Type":"ContainerDied","Data":"41f767cebc0e34d589b024bb3f00d006166e505bc6ab7177b4630eefddbeb831"} Oct 31 14:02:41.895796 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.895778 5532 scope.go:122] "RemoveContainer" containerID="07230cd7162ff00ec76cd832a1af6c9dcf3e7141c037c2303a60b0510752bb3e" Oct 31 14:02:41.896123 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:41.896100 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07230cd7162ff00ec76cd832a1af6c9dcf3e7141c037c2303a60b0510752bb3e\": container with ID starting with 07230cd7162ff00ec76cd832a1af6c9dcf3e7141c037c2303a60b0510752bb3e not found: ID does not exist" containerID="07230cd7162ff00ec76cd832a1af6c9dcf3e7141c037c2303a60b0510752bb3e" Oct 31 14:02:41.896251 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.896128 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07230cd7162ff00ec76cd832a1af6c9dcf3e7141c037c2303a60b0510752bb3e"} err="failed to get container status \"07230cd7162ff00ec76cd832a1af6c9dcf3e7141c037c2303a60b0510752bb3e\": rpc error: code = NotFound desc = could not find container \"07230cd7162ff00ec76cd832a1af6c9dcf3e7141c037c2303a60b0510752bb3e\": container with ID starting with 07230cd7162ff00ec76cd832a1af6c9dcf3e7141c037c2303a60b0510752bb3e not found: ID does not exist" Oct 31 14:02:41.896251 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.896146 5532 scope.go:122] "RemoveContainer" containerID="3dbe0f5ca021aa62c51d3ac34c28c4892636d8c2aa386ecef9a8ff9425b70615" Oct 31 14:02:41.908088 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.908069 5532 scope.go:122] "RemoveContainer" containerID="3dbe0f5ca021aa62c51d3ac34c28c4892636d8c2aa386ecef9a8ff9425b70615" Oct 31 14:02:41.908369 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:41.908350 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dbe0f5ca021aa62c51d3ac34c28c4892636d8c2aa386ecef9a8ff9425b70615\": container with ID starting with 3dbe0f5ca021aa62c51d3ac34c28c4892636d8c2aa386ecef9a8ff9425b70615 not found: ID does not exist" containerID="3dbe0f5ca021aa62c51d3ac34c28c4892636d8c2aa386ecef9a8ff9425b70615" Oct 31 14:02:41.908432 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.908379 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dbe0f5ca021aa62c51d3ac34c28c4892636d8c2aa386ecef9a8ff9425b70615"} err="failed to get container status \"3dbe0f5ca021aa62c51d3ac34c28c4892636d8c2aa386ecef9a8ff9425b70615\": rpc error: code = NotFound desc = could not find container \"3dbe0f5ca021aa62c51d3ac34c28c4892636d8c2aa386ecef9a8ff9425b70615\": container with ID starting with 3dbe0f5ca021aa62c51d3ac34c28c4892636d8c2aa386ecef9a8ff9425b70615 not found: ID does not exist" Oct 31 14:02:41.908432 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.908394 5532 scope.go:122] "RemoveContainer" containerID="8287b0a564d9f1f2384fc4d86efe6ff6a1e4ca093e0bdd1a2baccfc8fcbd1be0" Oct 31 14:02:41.917365 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.917341 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2"] Oct 31 14:02:41.921654 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.921118 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/simulator-w3-c82140-kserve-5867bc8964-sqxn2"] Oct 31 14:02:41.923636 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.923584 5532 scope.go:122] "RemoveContainer" containerID="8287b0a564d9f1f2384fc4d86efe6ff6a1e4ca093e0bdd1a2baccfc8fcbd1be0" Oct 31 14:02:41.924128 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:41.924105 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8287b0a564d9f1f2384fc4d86efe6ff6a1e4ca093e0bdd1a2baccfc8fcbd1be0\": container with ID starting with 8287b0a564d9f1f2384fc4d86efe6ff6a1e4ca093e0bdd1a2baccfc8fcbd1be0 not found: ID does not exist" containerID="8287b0a564d9f1f2384fc4d86efe6ff6a1e4ca093e0bdd1a2baccfc8fcbd1be0" Oct 31 14:02:41.924214 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.924136 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8287b0a564d9f1f2384fc4d86efe6ff6a1e4ca093e0bdd1a2baccfc8fcbd1be0"} err="failed to get container status \"8287b0a564d9f1f2384fc4d86efe6ff6a1e4ca093e0bdd1a2baccfc8fcbd1be0\": rpc error: code = NotFound desc = could not find container \"8287b0a564d9f1f2384fc4d86efe6ff6a1e4ca093e0bdd1a2baccfc8fcbd1be0\": container with ID starting with 8287b0a564d9f1f2384fc4d86efe6ff6a1e4ca093e0bdd1a2baccfc8fcbd1be0 not found: ID does not exist" Oct 31 14:02:41.933283 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.933260 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww"] Oct 31 14:02:41.940379 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.940358 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/distinct-w3-c82140-kserve-6b5f85765b-zp2ww"] Oct 31 14:02:41.953965 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.953941 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj"] Oct 31 14:02:41.958832 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.958803 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dccc053f-a13f-496e-8c6e-781f41397b58-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.958832 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.958827 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp24g\" (UniqueName: \"kubernetes.io/projected/dccc053f-a13f-496e-8c6e-781f41397b58-kube-api-access-tp24g\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.958979 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.958836 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dccc053f-a13f-496e-8c6e-781f41397b58-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:41.964218 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:41.964193 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c82140/distinct-2-w3-c82140-kserve-7b7fffc7b7-nrqtj"] Oct 31 14:02:42.002852 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:42.002775 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" path="/var/lib/kubelet/pods/09ab1bcc-1eac-475b-af39-23d43405e233/volumes" Oct 31 14:02:42.003317 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:42.003295 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" path="/var/lib/kubelet/pods/77ce8cc3-c32c-4441-be3c-4ca957f5c7be/volumes" Oct 31 14:02:42.003792 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:42.003773 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" path="/var/lib/kubelet/pods/dccc053f-a13f-496e-8c6e-781f41397b58/volumes" Oct 31 14:02:42.782110 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:42.782071 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" podUID="27513f81-9d1f-407b-bca8-86c3d9659bf0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:02:43.799286 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:43.799252 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" Oct 31 14:02:44.513982 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.513919 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" Oct 31 14:02:44.693619 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.693565 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27513f81-9d1f-407b-bca8-86c3d9659bf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27513f81-9d1f-407b-bca8-86c3d9659bf0-plugins-config-volume\") pod \"27513f81-9d1f-407b-bca8-86c3d9659bf0\" (UID: \"27513f81-9d1f-407b-bca8-86c3d9659bf0\") " Oct 31 14:02:44.693805 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.693676 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27513f81-9d1f-407b-bca8-86c3d9659bf0-kube-api-access-ndklm\" (UniqueName: \"kubernetes.io/projected/27513f81-9d1f-407b-bca8-86c3d9659bf0-kube-api-access-ndklm\") pod \"27513f81-9d1f-407b-bca8-86c3d9659bf0\" (UID: \"27513f81-9d1f-407b-bca8-86c3d9659bf0\") " Oct 31 14:02:44.694019 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.693993 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27513f81-9d1f-407b-bca8-86c3d9659bf0-plugins-config-volume" pod "27513f81-9d1f-407b-bca8-86c3d9659bf0" (UID: "27513f81-9d1f-407b-bca8-86c3d9659bf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:44.695778 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.695751 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27513f81-9d1f-407b-bca8-86c3d9659bf0-kube-api-access-ndklm" pod "27513f81-9d1f-407b-bca8-86c3d9659bf0" (UID: "27513f81-9d1f-407b-bca8-86c3d9659bf0"). InnerVolumeSpecName "kube-api-access-ndklm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:44.794736 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.794662 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27513f81-9d1f-407b-bca8-86c3d9659bf0-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:44.794736 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.794690 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndklm\" (UniqueName: \"kubernetes.io/projected/27513f81-9d1f-407b-bca8-86c3d9659bf0-kube-api-access-ndklm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:44.923644 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.923577 5532 generic.go:356] "Generic (PLEG): container finished" podID="27513f81-9d1f-407b-bca8-86c3d9659bf0" containerID="f79f349a648ce097da929ab5d5934688ada1e93099741d5d68cfebe59baea9af" exitCode=0 Oct 31 14:02:44.924177 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.923626 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" event={"ID":"27513f81-9d1f-407b-bca8-86c3d9659bf0","Type":"ContainerDied","Data":"f79f349a648ce097da929ab5d5934688ada1e93099741d5d68cfebe59baea9af"} Oct 31 14:02:44.924314 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.923672 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" Oct 31 14:02:44.924314 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.924247 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q" event={"ID":"27513f81-9d1f-407b-bca8-86c3d9659bf0","Type":"ContainerDied","Data":"ee5fb88a0e620b9d6ee05b61ca4d17c0d0aa2f09026240429374738ecda8b11a"} Oct 31 14:02:44.924314 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.924285 5532 scope.go:122] "RemoveContainer" containerID="f79f349a648ce097da929ab5d5934688ada1e93099741d5d68cfebe59baea9af" Oct 31 14:02:44.936574 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.936556 5532 scope.go:122] "RemoveContainer" containerID="f79f349a648ce097da929ab5d5934688ada1e93099741d5d68cfebe59baea9af" Oct 31 14:02:44.936867 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:44.936834 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f79f349a648ce097da929ab5d5934688ada1e93099741d5d68cfebe59baea9af\": container with ID starting with f79f349a648ce097da929ab5d5934688ada1e93099741d5d68cfebe59baea9af not found: ID does not exist" containerID="f79f349a648ce097da929ab5d5934688ada1e93099741d5d68cfebe59baea9af" Oct 31 14:02:44.936947 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.936878 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f79f349a648ce097da929ab5d5934688ada1e93099741d5d68cfebe59baea9af"} err="failed to get container status \"f79f349a648ce097da929ab5d5934688ada1e93099741d5d68cfebe59baea9af\": rpc error: code = NotFound desc = could not find container \"f79f349a648ce097da929ab5d5934688ada1e93099741d5d68cfebe59baea9af\": container with ID starting with f79f349a648ce097da929ab5d5934688ada1e93099741d5d68cfebe59baea9af not found: ID does not exist" Oct 31 14:02:44.955984 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.953843 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q"] Oct 31 14:02:44.957296 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:44.957270 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-6784d6-6dfd744559-gfw5q"] Oct 31 14:02:45.028939 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:45.028901 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd"] Oct 31 14:02:45.032802 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:45.032767 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-251c725c-qhhpd"] Oct 31 14:02:45.944538 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:45.944498 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5"] Oct 31 14:02:45.945002 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:45.944748 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" podUID="dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" containerName="istio-proxy" containerID="cri-o://284af228b9b6d4e7dec60804bc214244db3a6ad942bbb5e377bea114cdfb8f0d" gracePeriod=30 Oct 31 14:02:45.948425 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:45.948397 5532 generic.go:356] "Generic (PLEG): container finished" podID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerID="e2723632e4d3eb942d4650493f817764fc0f9bf7430f4cc4bbb3ea2d5feed0bb" exitCode=0 Oct 31 14:02:45.950775 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:45.950734 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" event={"ID":"6ca77a1a-0c6a-4264-bc01-4899ef1dad75","Type":"ContainerDied","Data":"e2723632e4d3eb942d4650493f817764fc0f9bf7430f4cc4bbb3ea2d5feed0bb"} Oct 31 14:02:45.951219 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:45.951200 5532 scope.go:122] "RemoveContainer" containerID="e2723632e4d3eb942d4650493f817764fc0f9bf7430f4cc4bbb3ea2d5feed0bb" Oct 31 14:02:45.953380 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:45.952904 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:02:45.953380 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:45.952991 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:02:45.953380 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:45.953037 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:02:46.005021 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:46.004997 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27513f81-9d1f-407b-bca8-86c3d9659bf0" path="/var/lib/kubelet/pods/27513f81-9d1f-407b-bca8-86c3d9659bf0/volumes" Oct 31 14:02:46.005450 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:46.005436 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="459a9c12-801e-456c-9da6-f64f297c9459" path="/var/lib/kubelet/pods/459a9c12-801e-456c-9da6-f64f297c9459/volumes" Oct 31 14:02:46.033270 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:46.033244 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8"] Oct 31 14:02:46.039995 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:46.039970 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8d9d1c3-kpfp8"] Oct 31 14:02:46.970969 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:46.970936 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" event={"ID":"6ca77a1a-0c6a-4264-bc01-4899ef1dad75","Type":"ContainerStarted","Data":"b1cd023168d13e2b3e5532e828c86c99796aa0b5387334dc06fe4408f82e0d09"} Oct 31 14:02:47.571294 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:47.571250 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:47.572658 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:47.572630 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Oct 31 14:02:47.572804 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:47.572680 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Oct 31 14:02:48.002287 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.002248 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719ab474-6de3-473c-8dd3-6240703d5de8" path="/var/lib/kubelet/pods/719ab474-6de3-473c-8dd3-6240703d5de8/volumes" Oct 31 14:02:48.188452 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.188420 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn"] Oct 31 14:02:48.188998 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.188981 5532 cpu_manager.go:414] "containerMap: removing container" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" Oct 31 14:02:48.189065 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.188999 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" Oct 31 14:02:48.189065 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.189027 5532 cpu_manager.go:414] "containerMap: removing container" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" Oct 31 14:02:48.189065 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.189033 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" Oct 31 14:02:48.189065 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.189051 5532 cpu_manager.go:414] "containerMap: removing container" podUID="27513f81-9d1f-407b-bca8-86c3d9659bf0" containerName="payload-processing" Oct 31 14:02:48.189065 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.189056 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27513f81-9d1f-407b-bca8-86c3d9659bf0" containerName="payload-processing" Oct 31 14:02:48.189236 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.189067 5532 cpu_manager.go:414] "containerMap: removing container" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" Oct 31 14:02:48.189236 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.189072 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" Oct 31 14:02:48.189236 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.189146 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccc053f-a13f-496e-8c6e-781f41397b58" containerName="main" Oct 31 14:02:48.189236 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.189158 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="27513f81-9d1f-407b-bca8-86c3d9659bf0" containerName="payload-processing" Oct 31 14:02:48.189236 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.189166 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ab1bcc-1eac-475b-af39-23d43405e233" containerName="main" Oct 31 14:02:48.189236 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.189176 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="77ce8cc3-c32c-4441-be3c-4ca957f5c7be" containerName="main" Oct 31 14:02:48.191750 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.191730 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" Oct 31 14:02:48.196647 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.196627 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:02:48.197300 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.197286 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:02:48.203950 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.203927 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn"] Oct 31 14:02:48.334888 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.334794 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cba2c3f-sdtcn\" (UID: \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" Oct 31 14:02:48.335070 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.334963 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cba2c3f-sdtcn\" (UID: \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" Oct 31 14:02:48.335152 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.335081 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzr9r\" (UniqueName: \"kubernetes.io/projected/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-kube-api-access-bzr9r\") pod \"maas-api-revoke-sub-1cba2c3f-sdtcn\" (UID: \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" Oct 31 14:02:48.435745 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.435703 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cba2c3f-sdtcn\" (UID: \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" Oct 31 14:02:48.435937 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.435811 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cba2c3f-sdtcn\" (UID: \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" Oct 31 14:02:48.435937 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.435883 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzr9r\" (UniqueName: \"kubernetes.io/projected/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-kube-api-access-bzr9r\") pod \"maas-api-revoke-sub-1cba2c3f-sdtcn\" (UID: \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" Oct 31 14:02:48.436458 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.436433 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1cba2c3f-sdtcn\" (UID: \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" Oct 31 14:02:48.438055 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.438027 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1cba2c3f-sdtcn\" (UID: \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" Oct 31 14:02:48.465319 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.465265 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzr9r\" (UniqueName: \"kubernetes.io/projected/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-kube-api-access-bzr9r\") pod \"maas-api-revoke-sub-1cba2c3f-sdtcn\" (UID: \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" Oct 31 14:02:48.469740 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.469708 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:48.505606 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.505571 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" Oct 31 14:02:48.573033 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.572995 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Oct 31 14:02:48.573270 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.573059 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Oct 31 14:02:48.828045 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.828012 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn"] Oct 31 14:02:48.830279 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:02:48.830240 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17552fd_3d5f_4e30_8f8a_90780d4c47b2.slice/crio-0ae6d92cbd8949c016cd575911af155de6f197dcd25e5edd8f2422c7b8dfc14d WatchSource:0}: Error finding container 0ae6d92cbd8949c016cd575911af155de6f197dcd25e5edd8f2422c7b8dfc14d: Status 404 returned error can't find the container with id 0ae6d92cbd8949c016cd575911af155de6f197dcd25e5edd8f2422c7b8dfc14d Oct 31 14:02:48.986305 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:48.986266 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" event={"ID":"a17552fd-3d5f-4e30-8f8a-90780d4c47b2","Type":"ContainerStarted","Data":"0ae6d92cbd8949c016cd575911af155de6f197dcd25e5edd8f2422c7b8dfc14d"} Oct 31 14:02:49.571890 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:49.571797 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Oct 31 14:02:49.571890 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:49.571851 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Oct 31 14:02:49.996130 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:49.996102 5532 generic.go:356] "Generic (PLEG): container finished" podID="a17552fd-3d5f-4e30-8f8a-90780d4c47b2" containerID="6297d21deadfd766a7a79645680530c0a03042acabc06ba77b69aa07df4ee2be" exitCode=0 Oct 31 14:02:50.003202 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:50.003171 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" event={"ID":"a17552fd-3d5f-4e30-8f8a-90780d4c47b2","Type":"ContainerDied","Data":"6297d21deadfd766a7a79645680530c0a03042acabc06ba77b69aa07df4ee2be"} Oct 31 14:02:50.463844 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:50.463785 5532 patch_prober.go:29] interesting pod/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:02:50.464041 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:50.463863 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" podUID="dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:02:50.571884 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:50.571853 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Oct 31 14:02:50.572328 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:50.571919 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Oct 31 14:02:51.006876 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.006836 5532 generic.go:356] "Generic (PLEG): container finished" podID="dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" containerID="284af228b9b6d4e7dec60804bc214244db3a6ad942bbb5e377bea114cdfb8f0d" exitCode=0 Oct 31 14:02:51.007275 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.007248 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" event={"ID":"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23","Type":"ContainerDied","Data":"284af228b9b6d4e7dec60804bc214244db3a6ad942bbb5e377bea114cdfb8f0d"} Oct 31 14:02:51.278684 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.278625 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 14:02:51.295713 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.295681 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-kube-api-access-62rgz\" (UniqueName: \"kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-kube-api-access-62rgz\") pod \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " Oct 31 14:02:51.295831 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.295769 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-certs\") pod \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " Oct 31 14:02:51.295831 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.295817 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-credential-socket\") pod \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " Oct 31 14:02:51.295915 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.295855 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-socket\") pod \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " Oct 31 14:02:51.295915 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.295885 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-token\" (UniqueName: \"kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-token\") pod \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " Oct 31 14:02:51.296022 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.295914 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-envoy\") pod \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " Oct 31 14:02:51.296022 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.295948 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istiod-ca-cert\") pod \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " Oct 31 14:02:51.296156 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.296098 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-certs" pod "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" (UID: "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:51.296225 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.296180 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-credential-socket" pod "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" (UID: "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:51.296225 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.296202 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-socket" pod "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" (UID: "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:51.296453 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.296430 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.296562 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.296463 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.296562 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.296478 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.296562 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.296435 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istiod-ca-cert" pod "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" (UID: "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:51.298130 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.298108 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-kube-api-access-62rgz" pod "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" (UID: "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23"). InnerVolumeSpecName "kube-api-access-62rgz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:51.298130 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.298116 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-envoy" pod "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" (UID: "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:51.298313 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.298289 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-token" pod "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" (UID: "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:51.303645 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.303626 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" Oct 31 14:02:51.397018 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.396966 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-xks-ca\") pod \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\" (UID: \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\") " Oct 31 14:02:51.397206 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.397083 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-kube-api-access-bzr9r\" (UniqueName: \"kubernetes.io/projected/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-kube-api-access-bzr9r\") pod \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\" (UID: \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\") " Oct 31 14:02:51.397206 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.397115 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-openshift-ca\") pod \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\" (UID: \"a17552fd-3d5f-4e30-8f8a-90780d4c47b2\") " Oct 31 14:02:51.397206 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.397185 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-podinfo\") pod \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " Oct 31 14:02:51.397411 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.397230 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-data\") pod \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\" (UID: \"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23\") " Oct 31 14:02:51.397554 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.397528 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-data" pod "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" (UID: "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:51.397635 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.397574 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-openshift-ca" pod "a17552fd-3d5f-4e30-8f8a-90780d4c47b2" (UID: "a17552fd-3d5f-4e30-8f8a-90780d4c47b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:02:51.397710 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.397634 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62rgz\" (UniqueName: \"kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-kube-api-access-62rgz\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.397710 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.397656 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.397710 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.397670 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.397710 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.397685 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.399263 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.399244 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-podinfo" pod "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" (UID: "dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:02:51.399666 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.399639 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-xks-ca" pod "a17552fd-3d5f-4e30-8f8a-90780d4c47b2" (UID: "a17552fd-3d5f-4e30-8f8a-90780d4c47b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:51.399666 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.399654 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-kube-api-access-bzr9r" pod "a17552fd-3d5f-4e30-8f8a-90780d4c47b2" (UID: "a17552fd-3d5f-4e30-8f8a-90780d4c47b2"). InnerVolumeSpecName "kube-api-access-bzr9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:51.499579 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.499544 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.499579 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.499580 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.499864 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.499615 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.499864 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.499630 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzr9r\" (UniqueName: \"kubernetes.io/projected/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-kube-api-access-bzr9r\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.499864 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.499645 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a17552fd-3d5f-4e30-8f8a-90780d4c47b2-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:51.572772 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.572189 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" start-of-body= Oct 31 14:02:51.572772 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:51.572249 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.67:15021/healthz/ready\": dial tcp 10.132.1.67:15021: connect: connection refused" Oct 31 14:02:52.015583 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.015560 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" Oct 31 14:02:52.015583 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.015566 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn" event={"ID":"a17552fd-3d5f-4e30-8f8a-90780d4c47b2","Type":"ContainerDied","Data":"0ae6d92cbd8949c016cd575911af155de6f197dcd25e5edd8f2422c7b8dfc14d"} Oct 31 14:02:52.015749 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.015655 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ae6d92cbd8949c016cd575911af155de6f197dcd25e5edd8f2422c7b8dfc14d" Oct 31 14:02:52.017421 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.017397 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" event={"ID":"dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23","Type":"ContainerDied","Data":"7a3ab1d34c212e7fc0c7a01d295b21856fdb12607a51e6076c8ff6c3492d5fed"} Oct 31 14:02:52.017611 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.017581 5532 scope.go:122] "RemoveContainer" containerID="284af228b9b6d4e7dec60804bc214244db3a6ad942bbb5e377bea114cdfb8f0d" Oct 31 14:02:52.018235 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.017491 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5" Oct 31 14:02:52.047059 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.047033 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5"] Oct 31 14:02:52.049376 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.049345 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-6784d6-openshift-default-5789cc64d5-nvnv5"] Oct 31 14:02:52.191943 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.191896 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" containerID="cri-o://8cc250dca7c653594d074a1ba0ff8b8ac53961d39d7899f965c2bd9084a9291f" gracePeriod=45 Oct 31 14:02:52.448767 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.448741 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 14:02:52.512037 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.510823 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tmp-dir\") pod \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " Oct 31 14:02:52.512037 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.510920 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-kube-api-access-mftx7\" (UniqueName: \"kubernetes.io/projected/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-kube-api-access-mftx7\") pod \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " Oct 31 14:02:52.512037 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.510981 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tls-certs\") pod \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " Oct 31 14:02:52.512037 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.511052 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-home\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-home\") pod \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " Oct 31 14:02:52.512037 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.511092 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-dshm\") pod \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " Oct 31 14:02:52.512037 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.511134 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-model-cache\") pod \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\" (UID: \"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7\") " Oct 31 14:02:52.512037 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.511248 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tmp-dir" pod "ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" (UID: "ad722dc1-2dd6-40bd-9a1d-818cf190d2f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:52.512037 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.511636 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:52.512037 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.511910 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-home" pod "ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" (UID: "ad722dc1-2dd6-40bd-9a1d-818cf190d2f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:52.513165 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.513136 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-model-cache" pod "ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" (UID: "ad722dc1-2dd6-40bd-9a1d-818cf190d2f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:52.514054 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.514028 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-kube-api-access-mftx7" pod "ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" (UID: "ad722dc1-2dd6-40bd-9a1d-818cf190d2f7"). InnerVolumeSpecName "kube-api-access-mftx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:02:52.514166 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.514062 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-dshm" pod "ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" (UID: "ad722dc1-2dd6-40bd-9a1d-818cf190d2f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:02:52.514323 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.514305 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tls-certs" pod "ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" (UID: "ad722dc1-2dd6-40bd-9a1d-818cf190d2f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:02:52.575851 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.575778 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:52.577140 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.577103 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:02:52.612903 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.612870 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:52.612903 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.612898 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:52.612903 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.612911 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:52.613195 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.612925 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:52.613195 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:52.612934 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mftx7\" (UniqueName: \"kubernetes.io/projected/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7-kube-api-access-mftx7\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:02:53.026493 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:53.026459 5532 generic.go:356] "Generic (PLEG): container finished" podID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerID="8cc250dca7c653594d074a1ba0ff8b8ac53961d39d7899f965c2bd9084a9291f" exitCode=0 Oct 31 14:02:53.026713 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:53.026497 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" Oct 31 14:02:53.026713 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:53.026544 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" event={"ID":"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7","Type":"ContainerDied","Data":"8cc250dca7c653594d074a1ba0ff8b8ac53961d39d7899f965c2bd9084a9291f"} Oct 31 14:02:53.026713 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:53.026585 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j" event={"ID":"ad722dc1-2dd6-40bd-9a1d-818cf190d2f7","Type":"ContainerDied","Data":"6e42b645e3b1da55e9f23b8c8a3bb7b9d64ebdb67aa718b9e58f4d2cb1ddfd39"} Oct 31 14:02:53.026713 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:53.026636 5532 scope.go:122] "RemoveContainer" containerID="8cc250dca7c653594d074a1ba0ff8b8ac53961d39d7899f965c2bd9084a9291f" Oct 31 14:02:53.038144 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:53.038121 5532 scope.go:122] "RemoveContainer" containerID="8cc250dca7c653594d074a1ba0ff8b8ac53961d39d7899f965c2bd9084a9291f" Oct 31 14:02:53.038412 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:02:53.038394 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cc250dca7c653594d074a1ba0ff8b8ac53961d39d7899f965c2bd9084a9291f\": container with ID starting with 8cc250dca7c653594d074a1ba0ff8b8ac53961d39d7899f965c2bd9084a9291f not found: ID does not exist" containerID="8cc250dca7c653594d074a1ba0ff8b8ac53961d39d7899f965c2bd9084a9291f" Oct 31 14:02:53.038465 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:53.038421 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cc250dca7c653594d074a1ba0ff8b8ac53961d39d7899f965c2bd9084a9291f"} err="failed to get container status \"8cc250dca7c653594d074a1ba0ff8b8ac53961d39d7899f965c2bd9084a9291f\": rpc error: code = NotFound desc = could not find container \"8cc250dca7c653594d074a1ba0ff8b8ac53961d39d7899f965c2bd9084a9291f\": container with ID starting with 8cc250dca7c653594d074a1ba0ff8b8ac53961d39d7899f965c2bd9084a9291f not found: ID does not exist" Oct 31 14:02:53.051770 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:53.051746 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j"] Oct 31 14:02:53.059122 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:53.059099 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-aabeef/simulator-w6-aabeef-kserve-698f45dd6f-8fh7j"] Oct 31 14:02:54.002937 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:54.002897 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" path="/var/lib/kubelet/pods/ad722dc1-2dd6-40bd-9a1d-818cf190d2f7/volumes" Oct 31 14:02:54.003396 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:54.003381 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" path="/var/lib/kubelet/pods/dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23/volumes" Oct 31 14:02:56.675710 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:56.675681 5532 scope.go:122] "RemoveContainer" containerID="e71eb27584eae16ed7146bc8886e912adc129c73d0884cf7a9a8486763b4314d" Oct 31 14:02:56.685538 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:56.685520 5532 scope.go:122] "RemoveContainer" containerID="a5cd7b8f43f685bcf846d023bc843cdb0581cd020ec1f68d1938410684e3ebbf" Oct 31 14:02:56.694869 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:56.694844 5532 scope.go:122] "RemoveContainer" containerID="e381d8f8ba184d66db3440dc15d5e73c0f2291ccc2326aafc1cbe8d044cfb17b" Oct 31 14:02:56.703841 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:56.703823 5532 scope.go:122] "RemoveContainer" containerID="323b94ff4cfd11c0d526402f90e94431fe6ba873b89612c5aea0867fe2333482" Oct 31 14:02:56.714004 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:02:56.713980 5532 scope.go:122] "RemoveContainer" containerID="75e5a16d934222f4eede96b40d988dfc26226b9dd414bc09829989b84cc6019b" Oct 31 14:03:00.322825 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:00.322779 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5"] Oct 31 14:03:00.323296 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:00.323038 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" podUID="e40c874b-6a3b-41db-8415-7db99e7dcf71" containerName="istio-proxy" containerID="cri-o://21906d21329a411f77572048159e5a3c7e43a292a620698652aa28189a990bc3" gracePeriod=30 Oct 31 14:03:01.220213 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:01.220178 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:03:01.761825 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:03:01.761788 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:03:05.582172 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.582149 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 14:03:05.662483 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.662448 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-certs\") pod \"e40c874b-6a3b-41db-8415-7db99e7dcf71\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " Oct 31 14:03:05.662676 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.662521 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-credential-socket\") pod \"e40c874b-6a3b-41db-8415-7db99e7dcf71\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " Oct 31 14:03:05.662676 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.662544 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e40c874b-6a3b-41db-8415-7db99e7dcf71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40c874b-6a3b-41db-8415-7db99e7dcf71-istiod-ca-cert\") pod \"e40c874b-6a3b-41db-8415-7db99e7dcf71\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " Oct 31 14:03:05.662676 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.662647 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-podinfo\") pod \"e40c874b-6a3b-41db-8415-7db99e7dcf71\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " Oct 31 14:03:05.662809 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.662678 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-socket\") pod \"e40c874b-6a3b-41db-8415-7db99e7dcf71\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " Oct 31 14:03:05.662851 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.662772 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-certs" pod "e40c874b-6a3b-41db-8415-7db99e7dcf71" (UID: "e40c874b-6a3b-41db-8415-7db99e7dcf71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:05.662851 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.662822 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-data\") pod \"e40c874b-6a3b-41db-8415-7db99e7dcf71\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " Oct 31 14:03:05.662969 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.662874 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-envoy\") pod \"e40c874b-6a3b-41db-8415-7db99e7dcf71\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " Oct 31 14:03:05.662969 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.662877 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-credential-socket" pod "e40c874b-6a3b-41db-8415-7db99e7dcf71" (UID: "e40c874b-6a3b-41db-8415-7db99e7dcf71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:05.662969 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.662907 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-kube-api-access-42c8c\" (UniqueName: \"kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-kube-api-access-42c8c\") pod \"e40c874b-6a3b-41db-8415-7db99e7dcf71\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " Oct 31 14:03:05.662969 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.662954 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-socket" pod "e40c874b-6a3b-41db-8415-7db99e7dcf71" (UID: "e40c874b-6a3b-41db-8415-7db99e7dcf71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:05.663226 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.663008 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-data" pod "e40c874b-6a3b-41db-8415-7db99e7dcf71" (UID: "e40c874b-6a3b-41db-8415-7db99e7dcf71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:05.663226 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.663021 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-token\" (UniqueName: \"kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-token\") pod \"e40c874b-6a3b-41db-8415-7db99e7dcf71\" (UID: \"e40c874b-6a3b-41db-8415-7db99e7dcf71\") " Oct 31 14:03:05.663226 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.663055 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e40c874b-6a3b-41db-8415-7db99e7dcf71-istiod-ca-cert" pod "e40c874b-6a3b-41db-8415-7db99e7dcf71" (UID: "e40c874b-6a3b-41db-8415-7db99e7dcf71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:03:05.663501 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.663475 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.663682 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.663505 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.663682 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.663522 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e40c874b-6a3b-41db-8415-7db99e7dcf71-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.663682 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.663536 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.663682 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.663551 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.665001 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.664961 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-podinfo" pod "e40c874b-6a3b-41db-8415-7db99e7dcf71" (UID: "e40c874b-6a3b-41db-8415-7db99e7dcf71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:03:05.665103 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.665026 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-envoy" pod "e40c874b-6a3b-41db-8415-7db99e7dcf71" (UID: "e40c874b-6a3b-41db-8415-7db99e7dcf71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:05.665103 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.665067 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-token" pod "e40c874b-6a3b-41db-8415-7db99e7dcf71" (UID: "e40c874b-6a3b-41db-8415-7db99e7dcf71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:05.665103 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.665089 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-kube-api-access-42c8c" pod "e40c874b-6a3b-41db-8415-7db99e7dcf71" (UID: "e40c874b-6a3b-41db-8415-7db99e7dcf71"). InnerVolumeSpecName "kube-api-access-42c8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:05.764659 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.764623 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.764659 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.764658 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.764945 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.764672 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e40c874b-6a3b-41db-8415-7db99e7dcf71-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:05.764945 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:05.764689 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42c8c\" (UniqueName: \"kubernetes.io/projected/e40c874b-6a3b-41db-8415-7db99e7dcf71-kube-api-access-42c8c\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:06.121813 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:06.121781 5532 generic.go:356] "Generic (PLEG): container finished" podID="e40c874b-6a3b-41db-8415-7db99e7dcf71" containerID="21906d21329a411f77572048159e5a3c7e43a292a620698652aa28189a990bc3" exitCode=0 Oct 31 14:03:06.122021 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:06.121832 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" Oct 31 14:03:06.122021 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:06.121869 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" event={"ID":"e40c874b-6a3b-41db-8415-7db99e7dcf71","Type":"ContainerDied","Data":"21906d21329a411f77572048159e5a3c7e43a292a620698652aa28189a990bc3"} Oct 31 14:03:06.122021 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:06.121933 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5" event={"ID":"e40c874b-6a3b-41db-8415-7db99e7dcf71","Type":"ContainerDied","Data":"0628a8e41a61180b0b91f6054eb30d25c181ad7a9d5f75cdca7fa18c925d3b32"} Oct 31 14:03:06.122021 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:06.121962 5532 scope.go:122] "RemoveContainer" containerID="21906d21329a411f77572048159e5a3c7e43a292a620698652aa28189a990bc3" Oct 31 14:03:06.132764 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:06.132745 5532 scope.go:122] "RemoveContainer" containerID="21906d21329a411f77572048159e5a3c7e43a292a620698652aa28189a990bc3" Oct 31 14:03:06.133028 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:03:06.133009 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21906d21329a411f77572048159e5a3c7e43a292a620698652aa28189a990bc3\": container with ID starting with 21906d21329a411f77572048159e5a3c7e43a292a620698652aa28189a990bc3 not found: ID does not exist" containerID="21906d21329a411f77572048159e5a3c7e43a292a620698652aa28189a990bc3" Oct 31 14:03:06.133077 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:06.133038 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21906d21329a411f77572048159e5a3c7e43a292a620698652aa28189a990bc3"} err="failed to get container status \"21906d21329a411f77572048159e5a3c7e43a292a620698652aa28189a990bc3\": rpc error: code = NotFound desc = could not find container \"21906d21329a411f77572048159e5a3c7e43a292a620698652aa28189a990bc3\": container with ID starting with 21906d21329a411f77572048159e5a3c7e43a292a620698652aa28189a990bc3 not found: ID does not exist" Oct 31 14:03:06.151632 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:06.151585 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5"] Oct 31 14:03:06.158919 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:06.158895 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c82140-openshift-default-6c6b7c6899-l8pj5"] Oct 31 14:03:08.002179 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:08.002141 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e40c874b-6a3b-41db-8415-7db99e7dcf71" path="/var/lib/kubelet/pods/e40c874b-6a3b-41db-8415-7db99e7dcf71/volumes" Oct 31 14:03:17.716498 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:17.716464 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:03:25.988747 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:25.988708 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n"] Oct 31 14:03:25.989288 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:25.988990 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" podUID="17d55c6c-187f-46d2-8994-cb0ef0412ca9" containerName="maas-api" containerID="cri-o://8fa460a4db75bc540b780e43dec01bc13909f28bca203fad0db86775edceed4e" gracePeriod=30 Oct 31 14:03:26.071790 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.071756 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz"] Oct 31 14:03:26.072052 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.072022 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" podUID="036c2383-7dbd-49ef-a30f-df6bf4d61820" containerName="payload-processing" containerID="cri-o://28e4e3e2df10503a8bfbbc8268c7adcadc4e72c42bd705abdb2d6c1cd18ec633" gracePeriod=30 Oct 31 14:03:26.257931 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.257857 5532 generic.go:356] "Generic (PLEG): container finished" podID="17d55c6c-187f-46d2-8994-cb0ef0412ca9" containerID="8fa460a4db75bc540b780e43dec01bc13909f28bca203fad0db86775edceed4e" exitCode=0 Oct 31 14:03:26.257931 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.257909 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" event={"ID":"17d55c6c-187f-46d2-8994-cb0ef0412ca9","Type":"ContainerDied","Data":"8fa460a4db75bc540b780e43dec01bc13909f28bca203fad0db86775edceed4e"} Oct 31 14:03:26.260132 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.260105 5532 generic.go:356] "Generic (PLEG): container finished" podID="036c2383-7dbd-49ef-a30f-df6bf4d61820" containerID="28e4e3e2df10503a8bfbbc8268c7adcadc4e72c42bd705abdb2d6c1cd18ec633" exitCode=0 Oct 31 14:03:26.260392 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.260311 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" event={"ID":"036c2383-7dbd-49ef-a30f-df6bf4d61820","Type":"ContainerDied","Data":"28e4e3e2df10503a8bfbbc8268c7adcadc4e72c42bd705abdb2d6c1cd18ec633"} Oct 31 14:03:26.357675 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.357651 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:03:26.374693 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.374665 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" Oct 31 14:03:26.384826 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.384792 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-maas-api-tls\") pod \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " Oct 31 14:03:26.385036 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.384925 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17d55c6c-187f-46d2-8994-cb0ef0412ca9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d55c6c-187f-46d2-8994-cb0ef0412ca9-openshift-service-ca\") pod \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " Oct 31 14:03:26.385036 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.385024 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17d55c6c-187f-46d2-8994-cb0ef0412ca9-kube-api-access-f5lfv\" (UniqueName: \"kubernetes.io/projected/17d55c6c-187f-46d2-8994-cb0ef0412ca9-kube-api-access-f5lfv\") pod \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " Oct 31 14:03:26.385213 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.385133 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-metrics-certs\") pod \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\" (UID: \"17d55c6c-187f-46d2-8994-cb0ef0412ca9\") " Oct 31 14:03:26.386084 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.386049 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17d55c6c-187f-46d2-8994-cb0ef0412ca9-openshift-service-ca" pod "17d55c6c-187f-46d2-8994-cb0ef0412ca9" (UID: "17d55c6c-187f-46d2-8994-cb0ef0412ca9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:03:26.389638 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.388841 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-metrics-certs" pod "17d55c6c-187f-46d2-8994-cb0ef0412ca9" (UID: "17d55c6c-187f-46d2-8994-cb0ef0412ca9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:03:26.389638 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.389222 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-maas-api-tls" pod "17d55c6c-187f-46d2-8994-cb0ef0412ca9" (UID: "17d55c6c-187f-46d2-8994-cb0ef0412ca9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:03:26.389638 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.389585 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d55c6c-187f-46d2-8994-cb0ef0412ca9-kube-api-access-f5lfv" pod "17d55c6c-187f-46d2-8994-cb0ef0412ca9" (UID: "17d55c6c-187f-46d2-8994-cb0ef0412ca9"). InnerVolumeSpecName "kube-api-access-f5lfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:26.486674 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.486618 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036c2383-7dbd-49ef-a30f-df6bf4d61820-kube-api-access-sqbsf\" (UniqueName: \"kubernetes.io/projected/036c2383-7dbd-49ef-a30f-df6bf4d61820-kube-api-access-sqbsf\") pod \"036c2383-7dbd-49ef-a30f-df6bf4d61820\" (UID: \"036c2383-7dbd-49ef-a30f-df6bf4d61820\") " Oct 31 14:03:26.486891 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.486689 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036c2383-7dbd-49ef-a30f-df6bf4d61820-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036c2383-7dbd-49ef-a30f-df6bf4d61820-plugins-config-volume\") pod \"036c2383-7dbd-49ef-a30f-df6bf4d61820\" (UID: \"036c2383-7dbd-49ef-a30f-df6bf4d61820\") " Oct 31 14:03:26.487167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.487136 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036c2383-7dbd-49ef-a30f-df6bf4d61820-plugins-config-volume" pod "036c2383-7dbd-49ef-a30f-df6bf4d61820" (UID: "036c2383-7dbd-49ef-a30f-df6bf4d61820"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:03:26.487257 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.487230 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/17d55c6c-187f-46d2-8994-cb0ef0412ca9-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:26.487327 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.487254 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5lfv\" (UniqueName: \"kubernetes.io/projected/17d55c6c-187f-46d2-8994-cb0ef0412ca9-kube-api-access-f5lfv\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:26.487327 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.487273 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:26.487327 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.487290 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/17d55c6c-187f-46d2-8994-cb0ef0412ca9-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:26.488847 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.488819 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036c2383-7dbd-49ef-a30f-df6bf4d61820-kube-api-access-sqbsf" pod "036c2383-7dbd-49ef-a30f-df6bf4d61820" (UID: "036c2383-7dbd-49ef-a30f-df6bf4d61820"). InnerVolumeSpecName "kube-api-access-sqbsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:26.588557 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.588479 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqbsf\" (UniqueName: \"kubernetes.io/projected/036c2383-7dbd-49ef-a30f-df6bf4d61820-kube-api-access-sqbsf\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:26.588557 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:26.588507 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/036c2383-7dbd-49ef-a30f-df6bf4d61820-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:27.267565 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:27.267418 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" Oct 31 14:03:27.268051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:27.267452 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n" event={"ID":"17d55c6c-187f-46d2-8994-cb0ef0412ca9","Type":"ContainerDied","Data":"af7dd1c6a1ddd942c533720af43bafb618b09da4ea766236173544c93e0dc5df"} Oct 31 14:03:27.268051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:27.267635 5532 scope.go:122] "RemoveContainer" containerID="8fa460a4db75bc540b780e43dec01bc13909f28bca203fad0db86775edceed4e" Oct 31 14:03:27.269181 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:27.269048 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" Oct 31 14:03:27.269181 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:27.269092 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz" event={"ID":"036c2383-7dbd-49ef-a30f-df6bf4d61820","Type":"ContainerDied","Data":"ca99322734ecebd82a49dc60d7205c732cd41406914a80fab6ed3a07d32c524e"} Oct 31 14:03:27.278282 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:27.278262 5532 scope.go:122] "RemoveContainer" containerID="28e4e3e2df10503a8bfbbc8268c7adcadc4e72c42bd705abdb2d6c1cd18ec633" Oct 31 14:03:27.303989 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:27.303965 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n"] Oct 31 14:03:27.308323 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:27.308299 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-1b75f9-84dbb59c45-hfb9n"] Oct 31 14:03:27.325537 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:27.325510 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz"] Oct 31 14:03:27.330493 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:27.330470 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1b75f9-57c6b6576d-p2kzz"] Oct 31 14:03:28.002778 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:28.002750 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036c2383-7dbd-49ef-a30f-df6bf4d61820" path="/var/lib/kubelet/pods/036c2383-7dbd-49ef-a30f-df6bf4d61820/volumes" Oct 31 14:03:28.003159 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:28.003146 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17d55c6c-187f-46d2-8994-cb0ef0412ca9" path="/var/lib/kubelet/pods/17d55c6c-187f-46d2-8994-cb0ef0412ca9/volumes" Oct 31 14:03:28.808926 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:28.808890 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:03:31.762791 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:03:31.762748 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:03:34.190315 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:34.190280 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj"] Oct 31 14:03:34.190862 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:34.190513 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerName="istio-proxy" containerID="cri-o://b1cd023168d13e2b3e5532e828c86c99796aa0b5387334dc06fe4408f82e0d09" gracePeriod=30 Oct 31 14:03:37.578311 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:37.578273 5532 patch_prober.go:29] interesting pod/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 14:03:37.578735 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:37.578325 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 14:03:39.352269 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.352230 5532 generic.go:356] "Generic (PLEG): container finished" podID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerID="b1cd023168d13e2b3e5532e828c86c99796aa0b5387334dc06fe4408f82e0d09" exitCode=0 Oct 31 14:03:39.352729 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.352295 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" event={"ID":"6ca77a1a-0c6a-4264-bc01-4899ef1dad75","Type":"ContainerDied","Data":"b1cd023168d13e2b3e5532e828c86c99796aa0b5387334dc06fe4408f82e0d09"} Oct 31 14:03:39.352729 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.352355 5532 scope.go:122] "RemoveContainer" containerID="e2723632e4d3eb942d4650493f817764fc0f9bf7430f4cc4bbb3ea2d5feed0bb" Oct 31 14:03:39.454695 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.454629 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:03:39.528538 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.528510 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-socket\") pod \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " Oct 31 14:03:39.528732 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.528553 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-certs\") pod \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " Oct 31 14:03:39.528732 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.528575 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-envoy\") pod \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " Oct 31 14:03:39.528732 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.528667 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istiod-ca-cert\") pod \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " Oct 31 14:03:39.528732 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.528717 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-kube-api-access-m5gm5\" (UniqueName: \"kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-kube-api-access-m5gm5\") pod \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " Oct 31 14:03:39.528924 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.528763 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-token\" (UniqueName: \"kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-token\") pod \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " Oct 31 14:03:39.528924 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.528805 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-socket" pod "6ca77a1a-0c6a-4264-bc01-4899ef1dad75" (UID: "6ca77a1a-0c6a-4264-bc01-4899ef1dad75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:39.528924 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.528818 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-data\") pod \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " Oct 31 14:03:39.528924 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.528858 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-certs" pod "6ca77a1a-0c6a-4264-bc01-4899ef1dad75" (UID: "6ca77a1a-0c6a-4264-bc01-4899ef1dad75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:39.528924 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.528882 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-podinfo\") pod \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " Oct 31 14:03:39.529213 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.528929 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-credential-socket\") pod \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\" (UID: \"6ca77a1a-0c6a-4264-bc01-4899ef1dad75\") " Oct 31 14:03:39.529213 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.529041 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-data" pod "6ca77a1a-0c6a-4264-bc01-4899ef1dad75" (UID: "6ca77a1a-0c6a-4264-bc01-4899ef1dad75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:39.529213 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.529123 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istiod-ca-cert" pod "6ca77a1a-0c6a-4264-bc01-4899ef1dad75" (UID: "6ca77a1a-0c6a-4264-bc01-4899ef1dad75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:03:39.529406 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.529317 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-credential-socket" pod "6ca77a1a-0c6a-4264-bc01-4899ef1dad75" (UID: "6ca77a1a-0c6a-4264-bc01-4899ef1dad75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:39.529491 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.529471 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:39.529553 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.529499 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:39.529553 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.529513 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:39.529553 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.529527 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:39.529553 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.529541 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:39.531023 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.530997 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-envoy" pod "6ca77a1a-0c6a-4264-bc01-4899ef1dad75" (UID: "6ca77a1a-0c6a-4264-bc01-4899ef1dad75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:39.531148 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.531059 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-kube-api-access-m5gm5" pod "6ca77a1a-0c6a-4264-bc01-4899ef1dad75" (UID: "6ca77a1a-0c6a-4264-bc01-4899ef1dad75"). InnerVolumeSpecName "kube-api-access-m5gm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:39.531148 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.531082 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-token" pod "6ca77a1a-0c6a-4264-bc01-4899ef1dad75" (UID: "6ca77a1a-0c6a-4264-bc01-4899ef1dad75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:39.531148 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.531097 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-podinfo" pod "6ca77a1a-0c6a-4264-bc01-4899ef1dad75" (UID: "6ca77a1a-0c6a-4264-bc01-4899ef1dad75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:03:39.630172 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.630133 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5gm5\" (UniqueName: \"kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-kube-api-access-m5gm5\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:39.630172 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.630167 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:39.630172 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.630177 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:39.630459 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.630186 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ca77a1a-0c6a-4264-bc01-4899ef1dad75-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:39.660546 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.660510 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567"] Oct 31 14:03:39.661135 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.661050 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" podUID="18e425c2-8c42-49f3-ba02-9e36514b53eb" containerName="discovery" containerID="cri-o://d3acd17ff8a44cdbadc70a106eb407e15f89b96ffd5db774a9758cf336002f7d" gracePeriod=30 Oct 31 14:03:39.910302 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:39.910276 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 14:03:40.033751 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.033723 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-kubeconfig\") pod \"18e425c2-8c42-49f3-ba02-9e36514b53eb\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " Oct 31 14:03:40.033880 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.033797 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e425c2-8c42-49f3-ba02-9e36514b53eb-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e425c2-8c42-49f3-ba02-9e36514b53eb-local-certs\") pod \"18e425c2-8c42-49f3-ba02-9e36514b53eb\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " Oct 31 14:03:40.033880 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.033839 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-dns-cert\") pod \"18e425c2-8c42-49f3-ba02-9e36514b53eb\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " Oct 31 14:03:40.033880 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.033865 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-ca-configmap\") pod \"18e425c2-8c42-49f3-ba02-9e36514b53eb\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " Oct 31 14:03:40.034053 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.033929 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-kube-api-access-lm9gg\" (UniqueName: \"kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-kube-api-access-lm9gg\") pod \"18e425c2-8c42-49f3-ba02-9e36514b53eb\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " Oct 31 14:03:40.034053 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.033970 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-cacerts\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-cacerts\") pod \"18e425c2-8c42-49f3-ba02-9e36514b53eb\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " Oct 31 14:03:40.034053 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.034000 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-token\" (UniqueName: \"kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-token\") pod \"18e425c2-8c42-49f3-ba02-9e36514b53eb\" (UID: \"18e425c2-8c42-49f3-ba02-9e36514b53eb\") " Oct 31 14:03:40.034495 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.034469 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-ca-configmap" pod "18e425c2-8c42-49f3-ba02-9e36514b53eb" (UID: "18e425c2-8c42-49f3-ba02-9e36514b53eb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:03:40.036492 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.036397 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-kube-api-access-lm9gg" pod "18e425c2-8c42-49f3-ba02-9e36514b53eb" (UID: "18e425c2-8c42-49f3-ba02-9e36514b53eb"). InnerVolumeSpecName "kube-api-access-lm9gg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:40.036611 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.036508 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-dns-cert" pod "18e425c2-8c42-49f3-ba02-9e36514b53eb" (UID: "18e425c2-8c42-49f3-ba02-9e36514b53eb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:03:40.036611 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.036522 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-kubeconfig" pod "18e425c2-8c42-49f3-ba02-9e36514b53eb" (UID: "18e425c2-8c42-49f3-ba02-9e36514b53eb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:03:40.036611 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.036538 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-cacerts" pod "18e425c2-8c42-49f3-ba02-9e36514b53eb" (UID: "18e425c2-8c42-49f3-ba02-9e36514b53eb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:03:40.036743 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.036679 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e425c2-8c42-49f3-ba02-9e36514b53eb-local-certs" pod "18e425c2-8c42-49f3-ba02-9e36514b53eb" (UID: "18e425c2-8c42-49f3-ba02-9e36514b53eb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:40.036743 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.036716 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-token" pod "18e425c2-8c42-49f3-ba02-9e36514b53eb" (UID: "18e425c2-8c42-49f3-ba02-9e36514b53eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:40.135750 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.135706 5532 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-kubeconfig\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:40.135750 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.135741 5532 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e425c2-8c42-49f3-ba02-9e36514b53eb-local-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:40.135750 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.135758 5532 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-dns-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:40.136064 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.135774 5532 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-csr-ca-configmap\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:40.136064 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.135791 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm9gg\" (UniqueName: \"kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-kube-api-access-lm9gg\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:40.136064 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.135806 5532 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/18e425c2-8c42-49f3-ba02-9e36514b53eb-cacerts\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:40.136064 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.135819 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e425c2-8c42-49f3-ba02-9e36514b53eb-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:40.359701 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.359620 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" Oct 31 14:03:40.360162 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.359619 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj" event={"ID":"6ca77a1a-0c6a-4264-bc01-4899ef1dad75","Type":"ContainerDied","Data":"c8a4d25183d37b534c3a4ba74410ad794769dd74fbe9db3200088abef37a6cd1"} Oct 31 14:03:40.360162 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.359826 5532 scope.go:122] "RemoveContainer" containerID="b1cd023168d13e2b3e5532e828c86c99796aa0b5387334dc06fe4408f82e0d09" Oct 31 14:03:40.361420 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.361396 5532 generic.go:356] "Generic (PLEG): container finished" podID="18e425c2-8c42-49f3-ba02-9e36514b53eb" containerID="d3acd17ff8a44cdbadc70a106eb407e15f89b96ffd5db774a9758cf336002f7d" exitCode=0 Oct 31 14:03:40.361485 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.361440 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" event={"ID":"18e425c2-8c42-49f3-ba02-9e36514b53eb","Type":"ContainerDied","Data":"d3acd17ff8a44cdbadc70a106eb407e15f89b96ffd5db774a9758cf336002f7d"} Oct 31 14:03:40.361485 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.361440 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" Oct 31 14:03:40.361485 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.361473 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567" event={"ID":"18e425c2-8c42-49f3-ba02-9e36514b53eb","Type":"ContainerDied","Data":"ed21b3e452a4e4d46f061aa91b49db656dd0efbd5ccfa2c94844328403e7d806"} Oct 31 14:03:40.370586 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.370566 5532 scope.go:122] "RemoveContainer" containerID="d3acd17ff8a44cdbadc70a106eb407e15f89b96ffd5db774a9758cf336002f7d" Oct 31 14:03:40.380305 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.380287 5532 scope.go:122] "RemoveContainer" containerID="d3acd17ff8a44cdbadc70a106eb407e15f89b96ffd5db774a9758cf336002f7d" Oct 31 14:03:40.380565 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:03:40.380548 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3acd17ff8a44cdbadc70a106eb407e15f89b96ffd5db774a9758cf336002f7d\": container with ID starting with d3acd17ff8a44cdbadc70a106eb407e15f89b96ffd5db774a9758cf336002f7d not found: ID does not exist" containerID="d3acd17ff8a44cdbadc70a106eb407e15f89b96ffd5db774a9758cf336002f7d" Oct 31 14:03:40.380628 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.380575 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3acd17ff8a44cdbadc70a106eb407e15f89b96ffd5db774a9758cf336002f7d"} err="failed to get container status \"d3acd17ff8a44cdbadc70a106eb407e15f89b96ffd5db774a9758cf336002f7d\": rpc error: code = NotFound desc = could not find container \"d3acd17ff8a44cdbadc70a106eb407e15f89b96ffd5db774a9758cf336002f7d\": container with ID starting with d3acd17ff8a44cdbadc70a106eb407e15f89b96ffd5db774a9758cf336002f7d not found: ID does not exist" Oct 31 14:03:40.390309 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.390285 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj"] Oct 31 14:03:40.398216 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.398194 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1b75f9-openshift-default-656d6dfd7c-zl9kj"] Oct 31 14:03:40.413813 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.413791 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567"] Oct 31 14:03:40.418237 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:40.418213 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-gk567"] Oct 31 14:03:42.002066 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:42.002032 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e425c2-8c42-49f3-ba02-9e36514b53eb" path="/var/lib/kubelet/pods/18e425c2-8c42-49f3-ba02-9e36514b53eb/volumes" Oct 31 14:03:42.002492 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:42.002478 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" path="/var/lib/kubelet/pods/6ca77a1a-0c6a-4264-bc01-4899ef1dad75/volumes" Oct 31 14:03:44.810738 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:44.810706 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk"] Oct 31 14:03:46.114542 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.114494 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms"] Oct 31 14:03:46.115092 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.114760 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" podUID="2536955f-80a6-4959-bc32-8b56a38f5631" containerName="maas-api" containerID="cri-o://91505def6b7265d15eb1cc06a334d41350be333c574c641c9284ccb693fb0440" gracePeriod=30 Oct 31 14:03:46.382532 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.382467 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 14:03:46.408860 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.408826 5532 generic.go:356] "Generic (PLEG): container finished" podID="2536955f-80a6-4959-bc32-8b56a38f5631" containerID="91505def6b7265d15eb1cc06a334d41350be333c574c641c9284ccb693fb0440" exitCode=0 Oct 31 14:03:46.409067 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.408881 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" Oct 31 14:03:46.409067 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.408966 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" event={"ID":"2536955f-80a6-4959-bc32-8b56a38f5631","Type":"ContainerDied","Data":"91505def6b7265d15eb1cc06a334d41350be333c574c641c9284ccb693fb0440"} Oct 31 14:03:46.409067 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.409015 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms" event={"ID":"2536955f-80a6-4959-bc32-8b56a38f5631","Type":"ContainerDied","Data":"a30d4cd7f7a78537f22a224848741085a7b1f7e0958c3bfb4a5f05888e903c86"} Oct 31 14:03:46.409067 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.409042 5532 scope.go:122] "RemoveContainer" containerID="91505def6b7265d15eb1cc06a334d41350be333c574c641c9284ccb693fb0440" Oct 31 14:03:46.419779 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.419759 5532 scope.go:122] "RemoveContainer" containerID="91505def6b7265d15eb1cc06a334d41350be333c574c641c9284ccb693fb0440" Oct 31 14:03:46.420438 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:03:46.420407 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91505def6b7265d15eb1cc06a334d41350be333c574c641c9284ccb693fb0440\": container with ID starting with 91505def6b7265d15eb1cc06a334d41350be333c574c641c9284ccb693fb0440 not found: ID does not exist" containerID="91505def6b7265d15eb1cc06a334d41350be333c574c641c9284ccb693fb0440" Oct 31 14:03:46.420559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.420450 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91505def6b7265d15eb1cc06a334d41350be333c574c641c9284ccb693fb0440"} err="failed to get container status \"91505def6b7265d15eb1cc06a334d41350be333c574c641c9284ccb693fb0440\": rpc error: code = NotFound desc = could not find container \"91505def6b7265d15eb1cc06a334d41350be333c574c641c9284ccb693fb0440\": container with ID starting with 91505def6b7265d15eb1cc06a334d41350be333c574c641c9284ccb693fb0440 not found: ID does not exist" Oct 31 14:03:46.502938 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.502907 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-maas-api-tls\") pod \"2536955f-80a6-4959-bc32-8b56a38f5631\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " Oct 31 14:03:46.503109 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.502954 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2536955f-80a6-4959-bc32-8b56a38f5631-kube-api-access-pthsq\" (UniqueName: \"kubernetes.io/projected/2536955f-80a6-4959-bc32-8b56a38f5631-kube-api-access-pthsq\") pod \"2536955f-80a6-4959-bc32-8b56a38f5631\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " Oct 31 14:03:46.503109 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.503057 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2536955f-80a6-4959-bc32-8b56a38f5631-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2536955f-80a6-4959-bc32-8b56a38f5631-openshift-service-ca\") pod \"2536955f-80a6-4959-bc32-8b56a38f5631\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " Oct 31 14:03:46.503248 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.503154 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-metrics-certs\") pod \"2536955f-80a6-4959-bc32-8b56a38f5631\" (UID: \"2536955f-80a6-4959-bc32-8b56a38f5631\") " Oct 31 14:03:46.503451 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.503422 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2536955f-80a6-4959-bc32-8b56a38f5631-openshift-service-ca" pod "2536955f-80a6-4959-bc32-8b56a38f5631" (UID: "2536955f-80a6-4959-bc32-8b56a38f5631"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:03:46.503536 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.503522 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2536955f-80a6-4959-bc32-8b56a38f5631-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:46.504930 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.504908 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-maas-api-tls" pod "2536955f-80a6-4959-bc32-8b56a38f5631" (UID: "2536955f-80a6-4959-bc32-8b56a38f5631"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:03:46.505420 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.505399 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-metrics-certs" pod "2536955f-80a6-4959-bc32-8b56a38f5631" (UID: "2536955f-80a6-4959-bc32-8b56a38f5631"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:03:46.505420 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.505411 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2536955f-80a6-4959-bc32-8b56a38f5631-kube-api-access-pthsq" pod "2536955f-80a6-4959-bc32-8b56a38f5631" (UID: "2536955f-80a6-4959-bc32-8b56a38f5631"). InnerVolumeSpecName "kube-api-access-pthsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:46.604806 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.604772 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:46.604806 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.604801 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2536955f-80a6-4959-bc32-8b56a38f5631-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:46.604806 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.604812 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pthsq\" (UniqueName: \"kubernetes.io/projected/2536955f-80a6-4959-bc32-8b56a38f5631-kube-api-access-pthsq\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:46.734122 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.734092 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms"] Oct 31 14:03:46.736643 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:46.736621 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-a39e7e-b8c677c58-8m4ms"] Oct 31 14:03:48.002679 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:48.002643 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2536955f-80a6-4959-bc32-8b56a38f5631" path="/var/lib/kubelet/pods/2536955f-80a6-4959-bc32-8b56a38f5631/volumes" Oct 31 14:03:48.901816 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:48.901783 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:03:49.811856 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:49.811814 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" podUID="8e4bae26-45a2-4f55-b30f-31fb08d18578" containerName="payload-processing" containerID="cri-o://b160164005b866ebc0335b943057fef8043c99ed1d5ba37a20fb3df43176fa55" gracePeriod=25 Oct 31 14:03:53.885710 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:53.885671 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" podUID="8e4bae26-45a2-4f55-b30f-31fb08d18578" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:03:54.286956 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.286916 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk"] Oct 31 14:03:54.287159 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.287134 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" podUID="b7361fc2-13c5-47db-ae28-cf18c0acfef6" containerName="istio-proxy" containerID="cri-o://06ba0522c955faef7a073a5f299d9dfc4ce110f43f5fd3cd2d479c87f5d6fe9b" gracePeriod=30 Oct 31 14:03:54.792224 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792183 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9"] Oct 31 14:03:54.792643 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792626 5532 cpu_manager.go:414] "containerMap: removing container" podUID="2536955f-80a6-4959-bc32-8b56a38f5631" containerName="maas-api" Oct 31 14:03:54.792707 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792646 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2536955f-80a6-4959-bc32-8b56a38f5631" containerName="maas-api" Oct 31 14:03:54.792707 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792667 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e40c874b-6a3b-41db-8415-7db99e7dcf71" containerName="istio-proxy" Oct 31 14:03:54.792707 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792672 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e40c874b-6a3b-41db-8415-7db99e7dcf71" containerName="istio-proxy" Oct 31 14:03:54.792707 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792683 5532 cpu_manager.go:414] "containerMap: removing container" podUID="17d55c6c-187f-46d2-8994-cb0ef0412ca9" containerName="maas-api" Oct 31 14:03:54.792707 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792689 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17d55c6c-187f-46d2-8994-cb0ef0412ca9" containerName="maas-api" Oct 31 14:03:54.792707 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792697 5532 cpu_manager.go:414] "containerMap: removing container" podUID="a17552fd-3d5f-4e30-8f8a-90780d4c47b2" containerName="revoke-keys" Oct 31 14:03:54.792707 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792702 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a17552fd-3d5f-4e30-8f8a-90780d4c47b2" containerName="revoke-keys" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792729 5532 cpu_manager.go:414] "containerMap: removing container" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerName="istio-proxy" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792735 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerName="istio-proxy" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792746 5532 cpu_manager.go:414] "containerMap: removing container" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerName="istio-proxy" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792752 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerName="istio-proxy" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792761 5532 cpu_manager.go:414] "containerMap: removing container" podUID="18e425c2-8c42-49f3-ba02-9e36514b53eb" containerName="discovery" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792769 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e425c2-8c42-49f3-ba02-9e36514b53eb" containerName="discovery" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792779 5532 cpu_manager.go:414] "containerMap: removing container" podUID="036c2383-7dbd-49ef-a30f-df6bf4d61820" containerName="payload-processing" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792786 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036c2383-7dbd-49ef-a30f-df6bf4d61820" containerName="payload-processing" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792804 5532 cpu_manager.go:414] "containerMap: removing container" podUID="dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" containerName="istio-proxy" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792810 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" containerName="istio-proxy" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792825 5532 cpu_manager.go:414] "containerMap: removing container" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792832 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792930 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="2536955f-80a6-4959-bc32-8b56a38f5631" containerName="maas-api" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792947 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e40c874b-6a3b-41db-8415-7db99e7dcf71" containerName="istio-proxy" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792957 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="17d55c6c-187f-46d2-8994-cb0ef0412ca9" containerName="maas-api" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792969 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="a17552fd-3d5f-4e30-8f8a-90780d4c47b2" containerName="revoke-keys" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792984 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerName="istio-proxy" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792993 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ca77a1a-0c6a-4264-bc01-4899ef1dad75" containerName="istio-proxy" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.792999 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e425c2-8c42-49f3-ba02-9e36514b53eb" containerName="discovery" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.793005 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="036c2383-7dbd-49ef-a30f-df6bf4d61820" containerName="payload-processing" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.793012 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbf59a63-8f9e-4fb5-acb3-2ca0cf6fba23" containerName="istio-proxy" Oct 31 14:03:54.793051 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.793018 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad722dc1-2dd6-40bd-9a1d-818cf190d2f7" containerName="main" Oct 31 14:03:54.796213 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.796197 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" Oct 31 14:03:54.804569 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.804544 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:03:54.804770 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.804750 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:03:54.807160 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.807135 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9"] Oct 31 14:03:54.994461 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.994418 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9\" (UID: \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" Oct 31 14:03:54.994937 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.994534 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9\" (UID: \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" Oct 31 14:03:54.994937 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:54.994579 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6jwg\" (UniqueName: \"kubernetes.io/projected/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-kube-api-access-b6jwg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9\" (UID: \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" Oct 31 14:03:55.095377 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:55.095277 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9\" (UID: \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" Oct 31 14:03:55.095668 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:55.095392 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9\" (UID: \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" Oct 31 14:03:55.095668 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:55.095456 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6jwg\" (UniqueName: \"kubernetes.io/projected/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-kube-api-access-b6jwg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9\" (UID: \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" Oct 31 14:03:55.096220 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:55.096199 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9\" (UID: \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" Oct 31 14:03:55.098036 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:55.098013 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9\" (UID: \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" Oct 31 14:03:55.109512 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:55.109487 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6jwg\" (UniqueName: \"kubernetes.io/projected/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-kube-api-access-b6jwg\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9\" (UID: \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" Oct 31 14:03:55.111324 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:55.111305 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" Oct 31 14:03:55.392271 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:55.392238 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9"] Oct 31 14:03:55.399251 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:03:55.399217 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f1ab2a_d1c4_4cd7_8ee3_e3a4e7b0a5e6.slice/crio-24a9628877a57bedab0fda32d80bd9603681dca24ce0c212a4331762a9d4b01f WatchSource:0}: Error finding container 24a9628877a57bedab0fda32d80bd9603681dca24ce0c212a4331762a9d4b01f: Status 404 returned error can't find the container with id 24a9628877a57bedab0fda32d80bd9603681dca24ce0c212a4331762a9d4b01f Oct 31 14:03:55.466380 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:55.466349 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" event={"ID":"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6","Type":"ContainerStarted","Data":"24a9628877a57bedab0fda32d80bd9603681dca24ce0c212a4331762a9d4b01f"} Oct 31 14:03:56.474161 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:56.474076 5532 generic.go:356] "Generic (PLEG): container finished" podID="f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6" containerID="e1ce1be704fc4bedaab993f01d6fb96fcfa6e631175c1a053d68c3a6c70daf0f" exitCode=0 Oct 31 14:03:56.474545 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:56.474162 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" event={"ID":"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6","Type":"ContainerDied","Data":"e1ce1be704fc4bedaab993f01d6fb96fcfa6e631175c1a053d68c3a6c70daf0f"} Oct 31 14:03:56.854831 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:56.854751 5532 scope.go:122] "RemoveContainer" containerID="07cb4f9b9c4f4a08a429d1005334cab2c59d062ab01cf01c7a32844e60350ea8" Oct 31 14:03:56.864643 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:56.864618 5532 scope.go:122] "RemoveContainer" containerID="c6f5213640633118891d437c61bd307e032ea7e9f82f91d9dc119b20e547fdf8" Oct 31 14:03:56.873027 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:56.873010 5532 scope.go:122] "RemoveContainer" containerID="6ca16a30f17d4c3f27a084f08bea64fed2547eb9ef5c90dc75c03d527703e672" Oct 31 14:03:56.881065 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:56.881043 5532 scope.go:122] "RemoveContainer" containerID="c6f3511d55390824e160a44129147d519db72cf5bdb4fca7ba89431606bb26d3" Oct 31 14:03:56.889054 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:56.889037 5532 scope.go:122] "RemoveContainer" containerID="2e6b0baaf11ca6922d735ec4ac3df64dfa17e77122cce0c0ecd380ea2a2103ec" Oct 31 14:03:56.897523 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:56.897503 5532 scope.go:122] "RemoveContainer" containerID="4d6f6adb0d400823dccc7f38bec3d4cd6355e91199507ddd479ce1775cb9abd1" Oct 31 14:03:56.905500 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:56.905483 5532 scope.go:122] "RemoveContainer" containerID="f866c326c265207da9168507f6156338724e8b96762e8b9fb765e8200b215701" Oct 31 14:03:57.623068 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:57.623044 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" Oct 31 14:03:57.718633 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:57.718583 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-openshift-ca\") pod \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\" (UID: \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\") " Oct 31 14:03:57.718791 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:57.718773 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-xks-ca\") pod \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\" (UID: \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\") " Oct 31 14:03:57.718890 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:57.718877 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-kube-api-access-b6jwg\" (UniqueName: \"kubernetes.io/projected/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-kube-api-access-b6jwg\") pod \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\" (UID: \"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6\") " Oct 31 14:03:57.719066 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:57.719038 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-openshift-ca" pod "f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6" (UID: "f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:03:57.719188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:57.719169 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:57.720726 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:57.720705 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-xks-ca" pod "f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6" (UID: "f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:03:57.721125 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:57.721093 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-kube-api-access-b6jwg" pod "f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6" (UID: "f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6"). InnerVolumeSpecName "kube-api-access-b6jwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:57.819866 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:57.819784 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:57.819866 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:57.819815 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6jwg\" (UniqueName: \"kubernetes.io/projected/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6-kube-api-access-b6jwg\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:58.486478 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:58.486447 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" Oct 31 14:03:58.486478 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:58.486459 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9" event={"ID":"f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6","Type":"ContainerDied","Data":"24a9628877a57bedab0fda32d80bd9603681dca24ce0c212a4331762a9d4b01f"} Oct 31 14:03:58.486743 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:58.486501 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24a9628877a57bedab0fda32d80bd9603681dca24ce0c212a4331762a9d4b01f" Oct 31 14:03:59.497189 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.497156 5532 generic.go:356] "Generic (PLEG): container finished" podID="b7361fc2-13c5-47db-ae28-cf18c0acfef6" containerID="06ba0522c955faef7a073a5f299d9dfc4ce110f43f5fd3cd2d479c87f5d6fe9b" exitCode=0 Oct 31 14:03:59.497662 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.497227 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" event={"ID":"b7361fc2-13c5-47db-ae28-cf18c0acfef6","Type":"ContainerDied","Data":"06ba0522c955faef7a073a5f299d9dfc4ce110f43f5fd3cd2d479c87f5d6fe9b"} Oct 31 14:03:59.542115 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.542093 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 14:03:59.635097 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635009 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-certs\") pod \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " Oct 31 14:03:59.635097 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635074 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-token\" (UniqueName: \"kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-token\") pod \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " Oct 31 14:03:59.635354 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635125 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-kube-api-access-75b85\" (UniqueName: \"kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-kube-api-access-75b85\") pod \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " Oct 31 14:03:59.635354 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635164 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-envoy\") pod \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " Oct 31 14:03:59.635354 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635209 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istiod-ca-cert\") pod \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " Oct 31 14:03:59.635354 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635237 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-socket\") pod \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " Oct 31 14:03:59.635354 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635270 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-podinfo\") pod \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " Oct 31 14:03:59.635354 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635326 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-data\") pod \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " Oct 31 14:03:59.635354 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635335 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-certs" pod "b7361fc2-13c5-47db-ae28-cf18c0acfef6" (UID: "b7361fc2-13c5-47db-ae28-cf18c0acfef6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:59.635810 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635416 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-credential-socket\") pod \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\" (UID: \"b7361fc2-13c5-47db-ae28-cf18c0acfef6\") " Oct 31 14:03:59.635810 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635557 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-socket" pod "b7361fc2-13c5-47db-ae28-cf18c0acfef6" (UID: "b7361fc2-13c5-47db-ae28-cf18c0acfef6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:59.635810 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635698 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istiod-ca-cert" pod "b7361fc2-13c5-47db-ae28-cf18c0acfef6" (UID: "b7361fc2-13c5-47db-ae28-cf18c0acfef6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:03:59.635810 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635796 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-credential-socket" pod "b7361fc2-13c5-47db-ae28-cf18c0acfef6" (UID: "b7361fc2-13c5-47db-ae28-cf18c0acfef6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:59.636100 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635926 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-data" pod "b7361fc2-13c5-47db-ae28-cf18c0acfef6" (UID: "b7361fc2-13c5-47db-ae28-cf18c0acfef6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:59.636100 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.635996 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:59.636100 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.636018 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:59.636100 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.636033 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:59.636100 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.636048 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:59.637580 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.637553 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-kube-api-access-75b85" pod "b7361fc2-13c5-47db-ae28-cf18c0acfef6" (UID: "b7361fc2-13c5-47db-ae28-cf18c0acfef6"). InnerVolumeSpecName "kube-api-access-75b85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:59.637733 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.637624 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-podinfo" pod "b7361fc2-13c5-47db-ae28-cf18c0acfef6" (UID: "b7361fc2-13c5-47db-ae28-cf18c0acfef6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:03:59.637844 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.637823 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-token" pod "b7361fc2-13c5-47db-ae28-cf18c0acfef6" (UID: "b7361fc2-13c5-47db-ae28-cf18c0acfef6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:03:59.637907 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.637862 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-envoy" pod "b7361fc2-13c5-47db-ae28-cf18c0acfef6" (UID: "b7361fc2-13c5-47db-ae28-cf18c0acfef6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:03:59.737618 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.737576 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75b85\" (UniqueName: \"kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-kube-api-access-75b85\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:59.737770 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.737624 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:59.737770 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.737636 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:59.737770 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.737645 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:03:59.737770 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:03:59.737653 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7361fc2-13c5-47db-ae28-cf18c0acfef6-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:00.513055 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:00.513022 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" Oct 31 14:04:00.513493 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:00.513050 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk" event={"ID":"b7361fc2-13c5-47db-ae28-cf18c0acfef6","Type":"ContainerDied","Data":"ba353cd63d51e06e9a513788640f7f1c29615039f225735f4c041829d39616e3"} Oct 31 14:04:00.513493 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:00.513151 5532 scope.go:122] "RemoveContainer" containerID="06ba0522c955faef7a073a5f299d9dfc4ce110f43f5fd3cd2d479c87f5d6fe9b" Oct 31 14:04:00.535944 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:00.535915 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk"] Oct 31 14:04:00.542549 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:00.542526 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-a39e7e-openshift-default-58476ffb8c-s6znk"] Oct 31 14:04:01.760701 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:04:01.760662 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:04:02.003351 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:02.003319 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7361fc2-13c5-47db-ae28-cf18c0acfef6" path="/var/lib/kubelet/pods/b7361fc2-13c5-47db-ae28-cf18c0acfef6/volumes" Oct 31 14:04:03.885357 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:03.885313 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" podUID="8e4bae26-45a2-4f55-b30f-31fb08d18578" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:04:06.593169 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.593138 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd"] Oct 31 14:04:06.593717 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.593341 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" podUID="4686420e-4a64-47bd-8f9b-c442fe8eda2d" containerName="maas-api" containerID="cri-o://71e924efb9b6f9d04540efba29bb80758e4ae04394f8d14c16e1dac748bb1d65" gracePeriod=30 Oct 31 14:04:06.692497 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.692458 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" podUID="4686420e-4a64-47bd-8f9b-c442fe8eda2d" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.221:8443/health\": dial tcp 10.132.0.221:8443: connect: connection refused" Oct 31 14:04:06.859142 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.859072 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 14:04:06.909586 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.909550 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4686420e-4a64-47bd-8f9b-c442fe8eda2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4686420e-4a64-47bd-8f9b-c442fe8eda2d-openshift-service-ca\") pod \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " Oct 31 14:04:06.909798 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.909667 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4686420e-4a64-47bd-8f9b-c442fe8eda2d-kube-api-access-jfbqs\" (UniqueName: \"kubernetes.io/projected/4686420e-4a64-47bd-8f9b-c442fe8eda2d-kube-api-access-jfbqs\") pod \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " Oct 31 14:04:06.909798 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.909720 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-metrics-certs\") pod \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " Oct 31 14:04:06.909798 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.909767 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-maas-api-tls\") pod \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\" (UID: \"4686420e-4a64-47bd-8f9b-c442fe8eda2d\") " Oct 31 14:04:06.910028 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.910002 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4686420e-4a64-47bd-8f9b-c442fe8eda2d-openshift-service-ca" pod "4686420e-4a64-47bd-8f9b-c442fe8eda2d" (UID: "4686420e-4a64-47bd-8f9b-c442fe8eda2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:04:06.910174 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.910142 5532 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4686420e-4a64-47bd-8f9b-c442fe8eda2d-openshift-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:06.911906 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.911873 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-maas-api-tls" pod "4686420e-4a64-47bd-8f9b-c442fe8eda2d" (UID: "4686420e-4a64-47bd-8f9b-c442fe8eda2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:04:06.912002 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.911889 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-metrics-certs" pod "4686420e-4a64-47bd-8f9b-c442fe8eda2d" (UID: "4686420e-4a64-47bd-8f9b-c442fe8eda2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:04:06.912002 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.911948 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4686420e-4a64-47bd-8f9b-c442fe8eda2d-kube-api-access-jfbqs" pod "4686420e-4a64-47bd-8f9b-c442fe8eda2d" (UID: "4686420e-4a64-47bd-8f9b-c442fe8eda2d"). InnerVolumeSpecName "kube-api-access-jfbqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:04:06.940180 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.940149 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9"] Oct 31 14:04:06.945972 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:06.945950 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-35d13e-mxhm9"] Oct 31 14:04:07.010857 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:07.010828 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:07.010857 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:07.010853 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4686420e-4a64-47bd-8f9b-c442fe8eda2d-maas-api-tls\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:07.011057 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:07.010865 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfbqs\" (UniqueName: \"kubernetes.io/projected/4686420e-4a64-47bd-8f9b-c442fe8eda2d-kube-api-access-jfbqs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:07.554210 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:07.554171 5532 generic.go:356] "Generic (PLEG): container finished" podID="4686420e-4a64-47bd-8f9b-c442fe8eda2d" containerID="71e924efb9b6f9d04540efba29bb80758e4ae04394f8d14c16e1dac748bb1d65" exitCode=0 Oct 31 14:04:07.554415 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:07.554220 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" Oct 31 14:04:07.554415 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:07.554250 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" event={"ID":"4686420e-4a64-47bd-8f9b-c442fe8eda2d","Type":"ContainerDied","Data":"71e924efb9b6f9d04540efba29bb80758e4ae04394f8d14c16e1dac748bb1d65"} Oct 31 14:04:07.554415 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:07.554313 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd" event={"ID":"4686420e-4a64-47bd-8f9b-c442fe8eda2d","Type":"ContainerDied","Data":"3e7489dd8e35e5c9a9cc0d928cb25b05cade99eb13b90f2f47afa3ed21c37432"} Oct 31 14:04:07.554415 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:07.554339 5532 scope.go:122] "RemoveContainer" containerID="71e924efb9b6f9d04540efba29bb80758e4ae04394f8d14c16e1dac748bb1d65" Oct 31 14:04:07.564511 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:07.564492 5532 scope.go:122] "RemoveContainer" containerID="71e924efb9b6f9d04540efba29bb80758e4ae04394f8d14c16e1dac748bb1d65" Oct 31 14:04:07.564793 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:04:07.564773 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71e924efb9b6f9d04540efba29bb80758e4ae04394f8d14c16e1dac748bb1d65\": container with ID starting with 71e924efb9b6f9d04540efba29bb80758e4ae04394f8d14c16e1dac748bb1d65 not found: ID does not exist" containerID="71e924efb9b6f9d04540efba29bb80758e4ae04394f8d14c16e1dac748bb1d65" Oct 31 14:04:07.564868 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:07.564805 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71e924efb9b6f9d04540efba29bb80758e4ae04394f8d14c16e1dac748bb1d65"} err="failed to get container status \"71e924efb9b6f9d04540efba29bb80758e4ae04394f8d14c16e1dac748bb1d65\": rpc error: code = NotFound desc = could not find container \"71e924efb9b6f9d04540efba29bb80758e4ae04394f8d14c16e1dac748bb1d65\": container with ID starting with 71e924efb9b6f9d04540efba29bb80758e4ae04394f8d14c16e1dac748bb1d65 not found: ID does not exist" Oct 31 14:04:07.578519 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:07.578481 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd"] Oct 31 14:04:07.584402 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:07.584377 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-35d13e-655ff696db-kfwfd"] Oct 31 14:04:08.001900 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:08.001869 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4686420e-4a64-47bd-8f9b-c442fe8eda2d" path="/var/lib/kubelet/pods/4686420e-4a64-47bd-8f9b-c442fe8eda2d/volumes" Oct 31 14:04:08.002304 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:08.002283 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6" path="/var/lib/kubelet/pods/f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6/volumes" Oct 31 14:04:09.298543 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:09.298508 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:04:10.069796 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.069737 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" Oct 31 14:04:10.141526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.141495 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4bae26-45a2-4f55-b30f-31fb08d18578-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4bae26-45a2-4f55-b30f-31fb08d18578-plugins-config-volume\") pod \"8e4bae26-45a2-4f55-b30f-31fb08d18578\" (UID: \"8e4bae26-45a2-4f55-b30f-31fb08d18578\") " Oct 31 14:04:10.141712 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.141544 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4bae26-45a2-4f55-b30f-31fb08d18578-kube-api-access-bqzvc\" (UniqueName: \"kubernetes.io/projected/8e4bae26-45a2-4f55-b30f-31fb08d18578-kube-api-access-bqzvc\") pod \"8e4bae26-45a2-4f55-b30f-31fb08d18578\" (UID: \"8e4bae26-45a2-4f55-b30f-31fb08d18578\") " Oct 31 14:04:10.141874 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.141852 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4bae26-45a2-4f55-b30f-31fb08d18578-plugins-config-volume" pod "8e4bae26-45a2-4f55-b30f-31fb08d18578" (UID: "8e4bae26-45a2-4f55-b30f-31fb08d18578"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:04:10.143691 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.143662 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4bae26-45a2-4f55-b30f-31fb08d18578-kube-api-access-bqzvc" pod "8e4bae26-45a2-4f55-b30f-31fb08d18578" (UID: "8e4bae26-45a2-4f55-b30f-31fb08d18578"). InnerVolumeSpecName "kube-api-access-bqzvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:04:10.243312 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.243282 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e4bae26-45a2-4f55-b30f-31fb08d18578-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:10.243312 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.243307 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqzvc\" (UniqueName: \"kubernetes.io/projected/8e4bae26-45a2-4f55-b30f-31fb08d18578-kube-api-access-bqzvc\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:10.575162 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.575126 5532 generic.go:356] "Generic (PLEG): container finished" podID="8e4bae26-45a2-4f55-b30f-31fb08d18578" containerID="b160164005b866ebc0335b943057fef8043c99ed1d5ba37a20fb3df43176fa55" exitCode=0 Oct 31 14:04:10.575672 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.575174 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" Oct 31 14:04:10.575672 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.575207 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" event={"ID":"8e4bae26-45a2-4f55-b30f-31fb08d18578","Type":"ContainerDied","Data":"b160164005b866ebc0335b943057fef8043c99ed1d5ba37a20fb3df43176fa55"} Oct 31 14:04:10.575672 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.575261 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk" event={"ID":"8e4bae26-45a2-4f55-b30f-31fb08d18578","Type":"ContainerDied","Data":"3765cfcd68f703576122400d7f4de665994995693218ed6ca9d36969571be3b3"} Oct 31 14:04:10.575672 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.575285 5532 scope.go:122] "RemoveContainer" containerID="b160164005b866ebc0335b943057fef8043c99ed1d5ba37a20fb3df43176fa55" Oct 31 14:04:10.588129 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.588110 5532 scope.go:122] "RemoveContainer" containerID="b160164005b866ebc0335b943057fef8043c99ed1d5ba37a20fb3df43176fa55" Oct 31 14:04:10.588408 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:04:10.588385 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b160164005b866ebc0335b943057fef8043c99ed1d5ba37a20fb3df43176fa55\": container with ID starting with b160164005b866ebc0335b943057fef8043c99ed1d5ba37a20fb3df43176fa55 not found: ID does not exist" containerID="b160164005b866ebc0335b943057fef8043c99ed1d5ba37a20fb3df43176fa55" Oct 31 14:04:10.588452 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.588418 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b160164005b866ebc0335b943057fef8043c99ed1d5ba37a20fb3df43176fa55"} err="failed to get container status \"b160164005b866ebc0335b943057fef8043c99ed1d5ba37a20fb3df43176fa55\": rpc error: code = NotFound desc = could not find container \"b160164005b866ebc0335b943057fef8043c99ed1d5ba37a20fb3df43176fa55\": container with ID starting with b160164005b866ebc0335b943057fef8043c99ed1d5ba37a20fb3df43176fa55 not found: ID does not exist" Oct 31 14:04:10.605385 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.605361 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk"] Oct 31 14:04:10.612098 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:10.612077 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-a39e7e-64f4c7ffdf-zfkfk"] Oct 31 14:04:12.001732 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:12.001699 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4bae26-45a2-4f55-b30f-31fb08d18578" path="/var/lib/kubelet/pods/8e4bae26-45a2-4f55-b30f-31fb08d18578/volumes" Oct 31 14:04:15.461859 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:15.461816 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s"] Oct 31 14:04:15.462307 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:15.462111 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" containerID="cri-o://7bbbca30ce49d48ccb4315e9c248a56eee0204e455a4868d99d8d153c627e582" gracePeriod=30 Oct 31 14:04:16.153548 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.153516 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht"] Oct 31 14:04:16.154007 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.153990 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6" containerName="revoke-keys" Oct 31 14:04:16.154080 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.154008 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6" containerName="revoke-keys" Oct 31 14:04:16.154080 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.154029 5532 cpu_manager.go:414] "containerMap: removing container" podUID="8e4bae26-45a2-4f55-b30f-31fb08d18578" containerName="payload-processing" Oct 31 14:04:16.154080 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.154034 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4bae26-45a2-4f55-b30f-31fb08d18578" containerName="payload-processing" Oct 31 14:04:16.154080 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.154046 5532 cpu_manager.go:414] "containerMap: removing container" podUID="4686420e-4a64-47bd-8f9b-c442fe8eda2d" containerName="maas-api" Oct 31 14:04:16.154080 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.154050 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4686420e-4a64-47bd-8f9b-c442fe8eda2d" containerName="maas-api" Oct 31 14:04:16.154080 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.154067 5532 cpu_manager.go:414] "containerMap: removing container" podUID="b7361fc2-13c5-47db-ae28-cf18c0acfef6" containerName="istio-proxy" Oct 31 14:04:16.154080 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.154074 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7361fc2-13c5-47db-ae28-cf18c0acfef6" containerName="istio-proxy" Oct 31 14:04:16.154343 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.154140 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7361fc2-13c5-47db-ae28-cf18c0acfef6" containerName="istio-proxy" Oct 31 14:04:16.154343 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.154151 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f1ab2a-d1c4-4cd7-8ee3-e3a4e7b0a5e6" containerName="revoke-keys" Oct 31 14:04:16.154343 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.154160 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4bae26-45a2-4f55-b30f-31fb08d18578" containerName="payload-processing" Oct 31 14:04:16.154343 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.154168 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="4686420e-4a64-47bd-8f9b-c442fe8eda2d" containerName="maas-api" Oct 31 14:04:16.158531 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.158513 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" Oct 31 14:04:16.165635 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.165487 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:04:16.165635 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.165525 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:04:16.165635 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.165588 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:04:16.165635 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.165616 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:04:16.167026 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.167003 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht"] Oct 31 14:04:16.207810 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.207777 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkg4p\" (UniqueName: \"kubernetes.io/projected/e84f2486-33f8-4e40-a547-365dbbd48e17-kube-api-access-bkg4p\") pod \"maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht\" (UID: \"e84f2486-33f8-4e40-a547-365dbbd48e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" Oct 31 14:04:16.208005 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.207815 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht\" (UID: \"e84f2486-33f8-4e40-a547-365dbbd48e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" Oct 31 14:04:16.208005 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.207892 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht\" (UID: \"e84f2486-33f8-4e40-a547-365dbbd48e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" Oct 31 14:04:16.309269 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.309237 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkg4p\" (UniqueName: \"kubernetes.io/projected/e84f2486-33f8-4e40-a547-365dbbd48e17-kube-api-access-bkg4p\") pod \"maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht\" (UID: \"e84f2486-33f8-4e40-a547-365dbbd48e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" Oct 31 14:04:16.309432 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.309272 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht\" (UID: \"e84f2486-33f8-4e40-a547-365dbbd48e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" Oct 31 14:04:16.309432 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.309309 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht\" (UID: \"e84f2486-33f8-4e40-a547-365dbbd48e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" Oct 31 14:04:16.309942 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.309922 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht\" (UID: \"e84f2486-33f8-4e40-a547-365dbbd48e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" Oct 31 14:04:16.311649 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.311626 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht\" (UID: \"e84f2486-33f8-4e40-a547-365dbbd48e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" Oct 31 14:04:16.322545 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.322518 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkg4p\" (UniqueName: \"kubernetes.io/projected/e84f2486-33f8-4e40-a547-365dbbd48e17-kube-api-access-bkg4p\") pod \"maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht\" (UID: \"e84f2486-33f8-4e40-a547-365dbbd48e17\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" Oct 31 14:04:16.473337 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.473259 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" Oct 31 14:04:16.749768 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:16.749744 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht"] Oct 31 14:04:16.752977 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:04:16.752940 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode84f2486_33f8_4e40_a547_365dbbd48e17.slice/crio-f603b239b5c56cbf796adecf08522155d5b56a3125e72b2541b83adaf6fb44c3 WatchSource:0}: Error finding container f603b239b5c56cbf796adecf08522155d5b56a3125e72b2541b83adaf6fb44c3: Status 404 returned error can't find the container with id f603b239b5c56cbf796adecf08522155d5b56a3125e72b2541b83adaf6fb44c3 Oct 31 14:04:17.618200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:17.618161 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" event={"ID":"e84f2486-33f8-4e40-a547-365dbbd48e17","Type":"ContainerStarted","Data":"2b1f3fa31c8593da77fd7fcb1ee8201210d8f54333677fbb0bb7ebd1311e8aad"} Oct 31 14:04:17.618805 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:17.618215 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" event={"ID":"e84f2486-33f8-4e40-a547-365dbbd48e17","Type":"ContainerStarted","Data":"f603b239b5c56cbf796adecf08522155d5b56a3125e72b2541b83adaf6fb44c3"} Oct 31 14:04:17.638818 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:17.638767 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" podStartSLOduration=1.638753131 podStartE2EDuration="1.638753131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:04:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:04:17.637186928 +0000 UTC m=+3625.782930758" watchObservedRunningTime="2026-10-31 14:04:17.638753131 +0000 UTC m=+3625.784496994" Oct 31 14:04:18.625702 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:18.625668 5532 generic.go:356] "Generic (PLEG): container finished" podID="e84f2486-33f8-4e40-a547-365dbbd48e17" containerID="2b1f3fa31c8593da77fd7fcb1ee8201210d8f54333677fbb0bb7ebd1311e8aad" exitCode=0 Oct 31 14:04:18.626131 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:18.625753 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" event={"ID":"e84f2486-33f8-4e40-a547-365dbbd48e17","Type":"ContainerDied","Data":"2b1f3fa31c8593da77fd7fcb1ee8201210d8f54333677fbb0bb7ebd1311e8aad"} Oct 31 14:04:19.774166 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:19.774137 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" Oct 31 14:04:19.854746 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:19.854713 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e84f2486-33f8-4e40-a547-365dbbd48e17-kube-api-access-bkg4p\" (UniqueName: \"kubernetes.io/projected/e84f2486-33f8-4e40-a547-365dbbd48e17-kube-api-access-bkg4p\") pod \"e84f2486-33f8-4e40-a547-365dbbd48e17\" (UID: \"e84f2486-33f8-4e40-a547-365dbbd48e17\") " Oct 31 14:04:19.854923 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:19.854791 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-xks-ca\") pod \"e84f2486-33f8-4e40-a547-365dbbd48e17\" (UID: \"e84f2486-33f8-4e40-a547-365dbbd48e17\") " Oct 31 14:04:19.854992 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:19.854931 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-openshift-ca\") pod \"e84f2486-33f8-4e40-a547-365dbbd48e17\" (UID: \"e84f2486-33f8-4e40-a547-365dbbd48e17\") " Oct 31 14:04:19.855278 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:19.855251 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-openshift-ca" pod "e84f2486-33f8-4e40-a547-365dbbd48e17" (UID: "e84f2486-33f8-4e40-a547-365dbbd48e17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:04:19.856895 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:19.856876 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-xks-ca" pod "e84f2486-33f8-4e40-a547-365dbbd48e17" (UID: "e84f2486-33f8-4e40-a547-365dbbd48e17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:04:19.857001 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:19.856899 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e84f2486-33f8-4e40-a547-365dbbd48e17-kube-api-access-bkg4p" pod "e84f2486-33f8-4e40-a547-365dbbd48e17" (UID: "e84f2486-33f8-4e40-a547-365dbbd48e17"). InnerVolumeSpecName "kube-api-access-bkg4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:04:19.956157 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:19.956110 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkg4p\" (UniqueName: \"kubernetes.io/projected/e84f2486-33f8-4e40-a547-365dbbd48e17-kube-api-access-bkg4p\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:19.956157 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:19.956153 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:19.956371 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:19.956164 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e84f2486-33f8-4e40-a547-365dbbd48e17-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:20.032644 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.032610 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht"] Oct 31 14:04:20.039246 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.039183 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs"] Oct 31 14:04:20.040747 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.040720 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-809457e8-vrxht"] Oct 31 14:04:20.042181 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.042156 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0890bf9-w97zs"] Oct 31 14:04:20.642668 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.642636 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" event={"ID":"e84f2486-33f8-4e40-a547-365dbbd48e17","Type":"ContainerDied","Data":"f603b239b5c56cbf796adecf08522155d5b56a3125e72b2541b83adaf6fb44c3"} Oct 31 14:04:20.642814 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.642699 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f603b239b5c56cbf796adecf08522155d5b56a3125e72b2541b83adaf6fb44c3" Oct 31 14:04:20.642814 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.642646 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht" Oct 31 14:04:20.645447 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.645422 5532 generic.go:356] "Generic (PLEG): container finished" podID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerID="7bbbca30ce49d48ccb4315e9c248a56eee0204e455a4868d99d8d153c627e582" exitCode=0 Oct 31 14:04:20.645587 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.645501 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" event={"ID":"d5eaae20-98cf-46cd-a44f-74d780a567cd","Type":"ContainerDied","Data":"7bbbca30ce49d48ccb4315e9c248a56eee0204e455a4868d99d8d153c627e582"} Oct 31 14:04:20.645587 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.645541 5532 scope.go:122] "RemoveContainer" containerID="26a440d1c1402f46d8873ef3a74b4f9df2ac0fa1b0258ecbad8bd84561da4f3f" Oct 31 14:04:20.706330 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.706305 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 14:04:20.865932 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.865893 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-socket\") pod \"d5eaae20-98cf-46cd-a44f-74d780a567cd\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " Oct 31 14:04:20.866526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.865954 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-certs\") pod \"d5eaae20-98cf-46cd-a44f-74d780a567cd\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " Oct 31 14:04:20.866526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.865988 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-podinfo\") pod \"d5eaae20-98cf-46cd-a44f-74d780a567cd\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " Oct 31 14:04:20.866526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866017 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-kube-api-access-wf8n9\" (UniqueName: \"kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-kube-api-access-wf8n9\") pod \"d5eaae20-98cf-46cd-a44f-74d780a567cd\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " Oct 31 14:04:20.866526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866053 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-envoy\") pod \"d5eaae20-98cf-46cd-a44f-74d780a567cd\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " Oct 31 14:04:20.866526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866079 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-token\" (UniqueName: \"kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-token\") pod \"d5eaae20-98cf-46cd-a44f-74d780a567cd\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " Oct 31 14:04:20.866526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866106 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-data\") pod \"d5eaae20-98cf-46cd-a44f-74d780a567cd\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " Oct 31 14:04:20.866526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866213 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5eaae20-98cf-46cd-a44f-74d780a567cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5eaae20-98cf-46cd-a44f-74d780a567cd-istiod-ca-cert\") pod \"d5eaae20-98cf-46cd-a44f-74d780a567cd\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " Oct 31 14:04:20.866526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866219 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-socket" pod "d5eaae20-98cf-46cd-a44f-74d780a567cd" (UID: "d5eaae20-98cf-46cd-a44f-74d780a567cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:04:20.866526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866293 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-credential-socket\") pod \"d5eaae20-98cf-46cd-a44f-74d780a567cd\" (UID: \"d5eaae20-98cf-46cd-a44f-74d780a567cd\") " Oct 31 14:04:20.866526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866353 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-certs" pod "d5eaae20-98cf-46cd-a44f-74d780a567cd" (UID: "d5eaae20-98cf-46cd-a44f-74d780a567cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:04:20.866526 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866507 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-data" pod "d5eaae20-98cf-46cd-a44f-74d780a567cd" (UID: "d5eaae20-98cf-46cd-a44f-74d780a567cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:04:20.867254 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866723 5532 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:20.867254 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866730 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5eaae20-98cf-46cd-a44f-74d780a567cd-istiod-ca-cert" pod "d5eaae20-98cf-46cd-a44f-74d780a567cd" (UID: "d5eaae20-98cf-46cd-a44f-74d780a567cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:04:20.867254 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866740 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-credential-socket" pod "d5eaae20-98cf-46cd-a44f-74d780a567cd" (UID: "d5eaae20-98cf-46cd-a44f-74d780a567cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:04:20.867254 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866745 5532 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-workload-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:20.867254 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.866786 5532 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-data\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:20.868447 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.868419 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-envoy" pod "d5eaae20-98cf-46cd-a44f-74d780a567cd" (UID: "d5eaae20-98cf-46cd-a44f-74d780a567cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:04:20.868447 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.868421 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-kube-api-access-wf8n9" pod "d5eaae20-98cf-46cd-a44f-74d780a567cd" (UID: "d5eaae20-98cf-46cd-a44f-74d780a567cd"). InnerVolumeSpecName "kube-api-access-wf8n9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:04:20.868640 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.868460 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-podinfo" pod "d5eaae20-98cf-46cd-a44f-74d780a567cd" (UID: "d5eaae20-98cf-46cd-a44f-74d780a567cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 14:04:20.868640 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.868579 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-token" pod "d5eaae20-98cf-46cd-a44f-74d780a567cd" (UID: "d5eaae20-98cf-46cd-a44f-74d780a567cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:04:20.934859 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.934828 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq"] Oct 31 14:04:20.967467 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.967432 5532 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5eaae20-98cf-46cd-a44f-74d780a567cd-istiod-ca-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:20.967467 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.967460 5532 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-credential-socket\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:20.967467 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.967470 5532 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-podinfo\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:20.967708 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.967479 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf8n9\" (UniqueName: \"kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-kube-api-access-wf8n9\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:20.967708 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.967490 5532 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-envoy\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:20.967708 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:20.967499 5532 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5eaae20-98cf-46cd-a44f-74d780a567cd-istio-token\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:21.652954 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:21.652918 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" event={"ID":"d5eaae20-98cf-46cd-a44f-74d780a567cd","Type":"ContainerDied","Data":"ae32d915ba89ef4f29224a94f742df6249d45257eb2eaa1b94995a8e0cb3adc8"} Oct 31 14:04:21.653164 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:21.652978 5532 scope.go:122] "RemoveContainer" containerID="7bbbca30ce49d48ccb4315e9c248a56eee0204e455a4868d99d8d153c627e582" Oct 31 14:04:21.653164 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:21.652980 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s" Oct 31 14:04:21.683924 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:21.683892 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s"] Oct 31 14:04:21.691539 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:21.691517 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-35d13e-openshift-default-cb5cd6b99-7n68s"] Oct 31 14:04:22.001493 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:22.001416 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7500c1d-c528-444e-8160-6ecf4244c146" path="/var/lib/kubelet/pods/c7500c1d-c528-444e-8160-6ecf4244c146/volumes" Oct 31 14:04:22.001886 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:22.001867 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" path="/var/lib/kubelet/pods/d5eaae20-98cf-46cd-a44f-74d780a567cd/volumes" Oct 31 14:04:22.002243 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:22.002229 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef9796f9-cb17-4345-98ff-e7de08c1309f" path="/var/lib/kubelet/pods/ef9796f9-cb17-4345-98ff-e7de08c1309f/volumes" Oct 31 14:04:26.203037 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.203004 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg"] Oct 31 14:04:26.331379 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.331345 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f"] Oct 31 14:04:26.331859 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.331836 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" Oct 31 14:04:26.331859 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.331856 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" Oct 31 14:04:26.332097 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.331883 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e84f2486-33f8-4e40-a547-365dbbd48e17" containerName="revoke-keys" Oct 31 14:04:26.332097 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.331892 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e84f2486-33f8-4e40-a547-365dbbd48e17" containerName="revoke-keys" Oct 31 14:04:26.332097 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.331906 5532 cpu_manager.go:414] "containerMap: removing container" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" Oct 31 14:04:26.332097 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.331914 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" Oct 31 14:04:26.332097 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.332062 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" Oct 31 14:04:26.332097 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.332085 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e84f2486-33f8-4e40-a547-365dbbd48e17" containerName="revoke-keys" Oct 31 14:04:26.335180 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.335161 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" Oct 31 14:04:26.345625 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.344293 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:04:26.345625 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.344305 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:04:26.345625 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.344361 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:04:26.345625 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.344489 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:04:26.346193 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.346172 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb"] Oct 31 14:04:26.346686 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.346668 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5eaae20-98cf-46cd-a44f-74d780a567cd" containerName="istio-proxy" Oct 31 14:04:26.351402 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.351381 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f"] Oct 31 14:04:26.351530 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.351508 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" Oct 31 14:04:26.363162 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.363136 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb"] Oct 31 14:04:26.526500 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.526412 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91f2bda2-bqt9f\" (UID: \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" Oct 31 14:04:26.526701 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.526499 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91f2bda2-bqt9f\" (UID: \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" Oct 31 14:04:26.526701 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.526668 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8fxv\" (UniqueName: \"kubernetes.io/projected/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-kube-api-access-p8fxv\") pod \"maas-api-revoke-sub-91f2bda2-bqt9f\" (UID: \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" Oct 31 14:04:26.526795 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.526705 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5dl5\" (UniqueName: \"kubernetes.io/projected/5d058e65-1d22-4861-9743-70b0ea3c2fad-kube-api-access-h5dl5\") pod \"maas-api-revoke-sub-ed40470d-vjtsb\" (UID: \"5d058e65-1d22-4861-9743-70b0ea3c2fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" Oct 31 14:04:26.526795 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.526733 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed40470d-vjtsb\" (UID: \"5d058e65-1d22-4861-9743-70b0ea3c2fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" Oct 31 14:04:26.526795 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.526791 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed40470d-vjtsb\" (UID: \"5d058e65-1d22-4861-9743-70b0ea3c2fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" Oct 31 14:04:26.627331 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.627289 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91f2bda2-bqt9f\" (UID: \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" Oct 31 14:04:26.627486 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.627341 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91f2bda2-bqt9f\" (UID: \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" Oct 31 14:04:26.627486 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.627404 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8fxv\" (UniqueName: \"kubernetes.io/projected/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-kube-api-access-p8fxv\") pod \"maas-api-revoke-sub-91f2bda2-bqt9f\" (UID: \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" Oct 31 14:04:26.627486 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.627426 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5dl5\" (UniqueName: \"kubernetes.io/projected/5d058e65-1d22-4861-9743-70b0ea3c2fad-kube-api-access-h5dl5\") pod \"maas-api-revoke-sub-ed40470d-vjtsb\" (UID: \"5d058e65-1d22-4861-9743-70b0ea3c2fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" Oct 31 14:04:26.627718 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.627486 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed40470d-vjtsb\" (UID: \"5d058e65-1d22-4861-9743-70b0ea3c2fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" Oct 31 14:04:26.627718 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.627528 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed40470d-vjtsb\" (UID: \"5d058e65-1d22-4861-9743-70b0ea3c2fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" Oct 31 14:04:26.628245 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.628224 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed40470d-vjtsb\" (UID: \"5d058e65-1d22-4861-9743-70b0ea3c2fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" Oct 31 14:04:26.628366 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.628245 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91f2bda2-bqt9f\" (UID: \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" Oct 31 14:04:26.629796 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.629777 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91f2bda2-bqt9f\" (UID: \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" Oct 31 14:04:26.630001 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.629980 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed40470d-vjtsb\" (UID: \"5d058e65-1d22-4861-9743-70b0ea3c2fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" Oct 31 14:04:26.640919 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.640898 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5dl5\" (UniqueName: \"kubernetes.io/projected/5d058e65-1d22-4861-9743-70b0ea3c2fad-kube-api-access-h5dl5\") pod \"maas-api-revoke-sub-ed40470d-vjtsb\" (UID: \"5d058e65-1d22-4861-9743-70b0ea3c2fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" Oct 31 14:04:26.641141 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.641118 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8fxv\" (UniqueName: \"kubernetes.io/projected/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-kube-api-access-p8fxv\") pod \"maas-api-revoke-sub-91f2bda2-bqt9f\" (UID: \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" Oct 31 14:04:26.649719 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.649700 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" Oct 31 14:04:26.665708 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.665685 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" Oct 31 14:04:26.933050 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.933024 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f"] Oct 31 14:04:26.935121 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:04:26.935076 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9d3729b_36e5_4ddd_ac81_ff4a7238254e.slice/crio-88ca6d7b3e09dd2a69f4f0271c274e2c5e2a96818b785a562d28b73e338a39a5 WatchSource:0}: Error finding container 88ca6d7b3e09dd2a69f4f0271c274e2c5e2a96818b785a562d28b73e338a39a5: Status 404 returned error can't find the container with id 88ca6d7b3e09dd2a69f4f0271c274e2c5e2a96818b785a562d28b73e338a39a5 Oct 31 14:04:26.959010 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:26.958989 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb"] Oct 31 14:04:26.966744 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:04:26.966718 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d058e65_1d22_4861_9743_70b0ea3c2fad.slice/crio-e12d4d9c8422b77391c866253d9abb46ec93476bf200184b25e6caf3c29bdb94 WatchSource:0}: Error finding container e12d4d9c8422b77391c866253d9abb46ec93476bf200184b25e6caf3c29bdb94: Status 404 returned error can't find the container with id e12d4d9c8422b77391c866253d9abb46ec93476bf200184b25e6caf3c29bdb94 Oct 31 14:04:27.691666 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:27.691631 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" event={"ID":"e9d3729b-36e5-4ddd-ac81-ff4a7238254e","Type":"ContainerStarted","Data":"4cc6167f4a52acb180ff28137da87dd5d8d1b31d86faa1444bf77dae7046fa90"} Oct 31 14:04:27.692096 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:27.691689 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" event={"ID":"e9d3729b-36e5-4ddd-ac81-ff4a7238254e","Type":"ContainerStarted","Data":"88ca6d7b3e09dd2a69f4f0271c274e2c5e2a96818b785a562d28b73e338a39a5"} Oct 31 14:04:27.693310 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:27.693282 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" event={"ID":"5d058e65-1d22-4861-9743-70b0ea3c2fad","Type":"ContainerStarted","Data":"66c332cb521624b5fefad7a28ac9b6307c516655dd5694443c0ded237a3cb043"} Oct 31 14:04:27.693447 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:27.693335 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" event={"ID":"5d058e65-1d22-4861-9743-70b0ea3c2fad","Type":"ContainerStarted","Data":"e12d4d9c8422b77391c866253d9abb46ec93476bf200184b25e6caf3c29bdb94"} Oct 31 14:04:27.718077 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:27.718034 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" podStartSLOduration=1.7180216860000002 podStartE2EDuration="1.718021686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:04:27.716004934 +0000 UTC m=+3635.861748764" watchObservedRunningTime="2026-10-31 14:04:27.718021686 +0000 UTC m=+3635.863765491" Oct 31 14:04:27.738391 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:27.738349 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" podStartSLOduration=1.73833839 podStartE2EDuration="1.73833839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:04:27.736277229 +0000 UTC m=+3635.882021058" watchObservedRunningTime="2026-10-31 14:04:27.73833839 +0000 UTC m=+3635.884082217" Oct 31 14:04:28.700688 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:28.700648 5532 generic.go:356] "Generic (PLEG): container finished" podID="e9d3729b-36e5-4ddd-ac81-ff4a7238254e" containerID="4cc6167f4a52acb180ff28137da87dd5d8d1b31d86faa1444bf77dae7046fa90" exitCode=0 Oct 31 14:04:28.701126 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:28.700737 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" event={"ID":"e9d3729b-36e5-4ddd-ac81-ff4a7238254e","Type":"ContainerDied","Data":"4cc6167f4a52acb180ff28137da87dd5d8d1b31d86faa1444bf77dae7046fa90"} Oct 31 14:04:28.702395 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:28.702361 5532 generic.go:356] "Generic (PLEG): container finished" podID="5d058e65-1d22-4861-9743-70b0ea3c2fad" containerID="66c332cb521624b5fefad7a28ac9b6307c516655dd5694443c0ded237a3cb043" exitCode=0 Oct 31 14:04:28.702544 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:28.702450 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" event={"ID":"5d058e65-1d22-4861-9743-70b0ea3c2fad","Type":"ContainerDied","Data":"66c332cb521624b5fefad7a28ac9b6307c516655dd5694443c0ded237a3cb043"} Oct 31 14:04:29.846530 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.846504 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" Oct 31 14:04:29.857281 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.857258 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-xks-ca\") pod \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\" (UID: \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\") " Oct 31 14:04:29.857405 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.857313 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-kube-api-access-p8fxv\" (UniqueName: \"kubernetes.io/projected/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-kube-api-access-p8fxv\") pod \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\" (UID: \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\") " Oct 31 14:04:29.857405 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.857379 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-openshift-ca\") pod \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\" (UID: \"e9d3729b-36e5-4ddd-ac81-ff4a7238254e\") " Oct 31 14:04:29.857806 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.857775 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-openshift-ca" pod "e9d3729b-36e5-4ddd-ac81-ff4a7238254e" (UID: "e9d3729b-36e5-4ddd-ac81-ff4a7238254e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:04:29.859564 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.859536 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-xks-ca" pod "e9d3729b-36e5-4ddd-ac81-ff4a7238254e" (UID: "e9d3729b-36e5-4ddd-ac81-ff4a7238254e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:04:29.859564 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.859548 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-kube-api-access-p8fxv" pod "e9d3729b-36e5-4ddd-ac81-ff4a7238254e" (UID: "e9d3729b-36e5-4ddd-ac81-ff4a7238254e"). InnerVolumeSpecName "kube-api-access-p8fxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:04:29.887968 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.887935 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" Oct 31 14:04:29.958855 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.958770 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-openshift-ca\") pod \"5d058e65-1d22-4861-9743-70b0ea3c2fad\" (UID: \"5d058e65-1d22-4861-9743-70b0ea3c2fad\") " Oct 31 14:04:29.958855 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.958826 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d058e65-1d22-4861-9743-70b0ea3c2fad-kube-api-access-h5dl5\" (UniqueName: \"kubernetes.io/projected/5d058e65-1d22-4861-9743-70b0ea3c2fad-kube-api-access-h5dl5\") pod \"5d058e65-1d22-4861-9743-70b0ea3c2fad\" (UID: \"5d058e65-1d22-4861-9743-70b0ea3c2fad\") " Oct 31 14:04:29.958855 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.958846 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-xks-ca\") pod \"5d058e65-1d22-4861-9743-70b0ea3c2fad\" (UID: \"5d058e65-1d22-4861-9743-70b0ea3c2fad\") " Oct 31 14:04:29.959180 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.959037 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:29.959180 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.959051 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:29.959180 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.959060 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8fxv\" (UniqueName: \"kubernetes.io/projected/e9d3729b-36e5-4ddd-ac81-ff4a7238254e-kube-api-access-p8fxv\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:29.959180 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.959150 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-openshift-ca" pod "5d058e65-1d22-4861-9743-70b0ea3c2fad" (UID: "5d058e65-1d22-4861-9743-70b0ea3c2fad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:04:29.960886 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.960860 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-xks-ca" pod "5d058e65-1d22-4861-9743-70b0ea3c2fad" (UID: "5d058e65-1d22-4861-9743-70b0ea3c2fad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:04:29.960986 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:29.960942 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d058e65-1d22-4861-9743-70b0ea3c2fad-kube-api-access-h5dl5" pod "5d058e65-1d22-4861-9743-70b0ea3c2fad" (UID: "5d058e65-1d22-4861-9743-70b0ea3c2fad"). InnerVolumeSpecName "kube-api-access-h5dl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:04:30.059619 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:30.059568 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:30.059619 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:30.059615 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5dl5\" (UniqueName: \"kubernetes.io/projected/5d058e65-1d22-4861-9743-70b0ea3c2fad-kube-api-access-h5dl5\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:30.059887 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:30.059631 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d058e65-1d22-4861-9743-70b0ea3c2fad-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:30.715257 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:30.715233 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" Oct 31 14:04:30.715419 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:30.715265 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb" event={"ID":"5d058e65-1d22-4861-9743-70b0ea3c2fad","Type":"ContainerDied","Data":"e12d4d9c8422b77391c866253d9abb46ec93476bf200184b25e6caf3c29bdb94"} Oct 31 14:04:30.715419 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:30.715324 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e12d4d9c8422b77391c866253d9abb46ec93476bf200184b25e6caf3c29bdb94" Oct 31 14:04:30.716830 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:30.716811 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" Oct 31 14:04:30.716969 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:30.716833 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f" event={"ID":"e9d3729b-36e5-4ddd-ac81-ff4a7238254e","Type":"ContainerDied","Data":"88ca6d7b3e09dd2a69f4f0271c274e2c5e2a96818b785a562d28b73e338a39a5"} Oct 31 14:04:30.716969 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:30.716880 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88ca6d7b3e09dd2a69f4f0271c274e2c5e2a96818b785a562d28b73e338a39a5" Oct 31 14:04:31.204227 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:31.204184 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" podUID="e44283dd-c621-4479-bbd2-b392cfe2a176" containerName="payload-pre-processing" containerID="cri-o://180b7629250c6670a21424e2b7be4e63d24d6b4d6e276b1118a7d616f1f2584a" gracePeriod=25 Oct 31 14:04:31.760614 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:04:31.760554 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:04:34.793459 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:34.793421 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" podUID="e44283dd-c621-4479-bbd2-b392cfe2a176" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:04:35.954829 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:35.954786 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" containerID="cri-o://916beffa560feb0154ae2c082af496056624a6f1f28de9f8e8edfcc08c86a93b" gracePeriod=45 Oct 31 14:04:36.030303 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.030263 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.74:8000/ready\": dial tcp 10.132.0.74:8000: connect: connection refused" Oct 31 14:04:36.203943 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.203919 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 14:04:36.219840 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.219782 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-dshm\") pod \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " Oct 31 14:04:36.219840 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.219823 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27dbfb95-a109-4e31-b1a0-4ac435c252d3-kube-api-access-8n48g\" (UniqueName: \"kubernetes.io/projected/27dbfb95-a109-4e31-b1a0-4ac435c252d3-kube-api-access-8n48g\") pod \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " Oct 31 14:04:36.220030 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.219925 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-model-cache\") pod \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " Oct 31 14:04:36.220030 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.220016 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-home\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-home\") pod \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " Oct 31 14:04:36.220187 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.220167 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-model-cache" pod "27dbfb95-a109-4e31-b1a0-4ac435c252d3" (UID: "27dbfb95-a109-4e31-b1a0-4ac435c252d3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:04:36.220289 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.220267 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-home" pod "27dbfb95-a109-4e31-b1a0-4ac435c252d3" (UID: "27dbfb95-a109-4e31-b1a0-4ac435c252d3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:04:36.220463 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.220445 5532 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-model-cache\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:36.220463 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.220464 5532 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-home\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:36.221925 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.221896 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-dshm" pod "27dbfb95-a109-4e31-b1a0-4ac435c252d3" (UID: "27dbfb95-a109-4e31-b1a0-4ac435c252d3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:04:36.222020 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.221973 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27dbfb95-a109-4e31-b1a0-4ac435c252d3-kube-api-access-8n48g" pod "27dbfb95-a109-4e31-b1a0-4ac435c252d3" (UID: "27dbfb95-a109-4e31-b1a0-4ac435c252d3"). InnerVolumeSpecName "kube-api-access-8n48g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:04:36.321177 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.321145 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tmp-dir\") pod \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " Oct 31 14:04:36.321369 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.321192 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tls-certs\" (UniqueName: \"kubernetes.io/secret/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tls-certs\") pod \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\" (UID: \"27dbfb95-a109-4e31-b1a0-4ac435c252d3\") " Oct 31 14:04:36.321499 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.321468 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tmp-dir" pod "27dbfb95-a109-4e31-b1a0-4ac435c252d3" (UID: "27dbfb95-a109-4e31-b1a0-4ac435c252d3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 14:04:36.321758 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.321734 5532 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-dshm\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:36.321812 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.321768 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n48g\" (UniqueName: \"kubernetes.io/projected/27dbfb95-a109-4e31-b1a0-4ac435c252d3-kube-api-access-8n48g\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:36.321812 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.321787 5532 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tmp-dir\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:36.323350 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.323327 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tls-certs" pod "27dbfb95-a109-4e31-b1a0-4ac435c252d3" (UID: "27dbfb95-a109-4e31-b1a0-4ac435c252d3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:04:36.423173 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.423142 5532 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27dbfb95-a109-4e31-b1a0-4ac435c252d3-tls-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:36.757247 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.757204 5532 generic.go:356] "Generic (PLEG): container finished" podID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerID="916beffa560feb0154ae2c082af496056624a6f1f28de9f8e8edfcc08c86a93b" exitCode=0 Oct 31 14:04:36.757521 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.757352 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" event={"ID":"27dbfb95-a109-4e31-b1a0-4ac435c252d3","Type":"ContainerDied","Data":"916beffa560feb0154ae2c082af496056624a6f1f28de9f8e8edfcc08c86a93b"} Oct 31 14:04:36.757521 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.757389 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" event={"ID":"27dbfb95-a109-4e31-b1a0-4ac435c252d3","Type":"ContainerDied","Data":"0ecf6b89f9a9172c2079b4da536c1f2b2b75af47c2b6e52d04b2a509e5b644f8"} Oct 31 14:04:36.757521 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.757420 5532 scope.go:122] "RemoveContainer" containerID="916beffa560feb0154ae2c082af496056624a6f1f28de9f8e8edfcc08c86a93b" Oct 31 14:04:36.758033 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.757855 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq" Oct 31 14:04:36.769961 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.769942 5532 scope.go:122] "RemoveContainer" containerID="916beffa560feb0154ae2c082af496056624a6f1f28de9f8e8edfcc08c86a93b" Oct 31 14:04:36.770278 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:04:36.770244 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"916beffa560feb0154ae2c082af496056624a6f1f28de9f8e8edfcc08c86a93b\": container with ID starting with 916beffa560feb0154ae2c082af496056624a6f1f28de9f8e8edfcc08c86a93b not found: ID does not exist" containerID="916beffa560feb0154ae2c082af496056624a6f1f28de9f8e8edfcc08c86a93b" Oct 31 14:04:36.770321 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.770288 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"916beffa560feb0154ae2c082af496056624a6f1f28de9f8e8edfcc08c86a93b"} err="failed to get container status \"916beffa560feb0154ae2c082af496056624a6f1f28de9f8e8edfcc08c86a93b\": rpc error: code = NotFound desc = could not find container \"916beffa560feb0154ae2c082af496056624a6f1f28de9f8e8edfcc08c86a93b\": container with ID starting with 916beffa560feb0154ae2c082af496056624a6f1f28de9f8e8edfcc08c86a93b not found: ID does not exist" Oct 31 14:04:36.785356 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.785330 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq"] Oct 31 14:04:36.790281 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:36.790249 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-8b2f0a/distinct-2-w4-8b2f0a-kserve-85f6687747-p6wjq"] Oct 31 14:04:38.002173 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:38.002136 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" path="/var/lib/kubelet/pods/27dbfb95-a109-4e31-b1a0-4ac435c252d3/volumes" Oct 31 14:04:38.674867 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:38.674829 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:04:41.628560 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:41.628525 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht"] Oct 31 14:04:41.634456 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:41.634428 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-8b2f0a-t9dht"] Oct 31 14:04:42.002960 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:42.002928 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e84f2486-33f8-4e40-a547-365dbbd48e17" path="/var/lib/kubelet/pods/e84f2486-33f8-4e40-a547-365dbbd48e17/volumes" Oct 31 14:04:44.480630 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:44.480576 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:04:44.793895 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:44.793813 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" podUID="e44283dd-c621-4479-bbd2-b392cfe2a176" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 14:04:51.450148 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.450121 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" Oct 31 14:04:51.565369 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.565285 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e44283dd-c621-4479-bbd2-b392cfe2a176-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e44283dd-c621-4479-bbd2-b392cfe2a176-plugins-config-volume\") pod \"e44283dd-c621-4479-bbd2-b392cfe2a176\" (UID: \"e44283dd-c621-4479-bbd2-b392cfe2a176\") " Oct 31 14:04:51.565369 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.565349 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e44283dd-c621-4479-bbd2-b392cfe2a176-kube-api-access-92s5n\" (UniqueName: \"kubernetes.io/projected/e44283dd-c621-4479-bbd2-b392cfe2a176-kube-api-access-92s5n\") pod \"e44283dd-c621-4479-bbd2-b392cfe2a176\" (UID: \"e44283dd-c621-4479-bbd2-b392cfe2a176\") " Oct 31 14:04:51.565772 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.565740 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e44283dd-c621-4479-bbd2-b392cfe2a176-plugins-config-volume" pod "e44283dd-c621-4479-bbd2-b392cfe2a176" (UID: "e44283dd-c621-4479-bbd2-b392cfe2a176"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:04:51.565840 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.565816 5532 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e44283dd-c621-4479-bbd2-b392cfe2a176-plugins-config-volume\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:51.567424 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.567402 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e44283dd-c621-4479-bbd2-b392cfe2a176-kube-api-access-92s5n" pod "e44283dd-c621-4479-bbd2-b392cfe2a176" (UID: "e44283dd-c621-4479-bbd2-b392cfe2a176"). InnerVolumeSpecName "kube-api-access-92s5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:04:51.667245 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.667215 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92s5n\" (UniqueName: \"kubernetes.io/projected/e44283dd-c621-4479-bbd2-b392cfe2a176-kube-api-access-92s5n\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:04:51.844789 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.844697 5532 generic.go:356] "Generic (PLEG): container finished" podID="e44283dd-c621-4479-bbd2-b392cfe2a176" containerID="180b7629250c6670a21424e2b7be4e63d24d6b4d6e276b1118a7d616f1f2584a" exitCode=0 Oct 31 14:04:51.844789 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.844753 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" Oct 31 14:04:51.844996 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.844778 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" event={"ID":"e44283dd-c621-4479-bbd2-b392cfe2a176","Type":"ContainerDied","Data":"180b7629250c6670a21424e2b7be4e63d24d6b4d6e276b1118a7d616f1f2584a"} Oct 31 14:04:51.844996 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.844835 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg" event={"ID":"e44283dd-c621-4479-bbd2-b392cfe2a176","Type":"ContainerDied","Data":"7a16c0ca2dfe92930cb3c4b29d82a5bba6e2026969ecfa9bcd8b8682fa60deb0"} Oct 31 14:04:51.844996 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.844858 5532 scope.go:122] "RemoveContainer" containerID="180b7629250c6670a21424e2b7be4e63d24d6b4d6e276b1118a7d616f1f2584a" Oct 31 14:04:51.854538 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.854522 5532 scope.go:122] "RemoveContainer" containerID="180b7629250c6670a21424e2b7be4e63d24d6b4d6e276b1118a7d616f1f2584a" Oct 31 14:04:51.854828 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:04:51.854809 5532 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"180b7629250c6670a21424e2b7be4e63d24d6b4d6e276b1118a7d616f1f2584a\": container with ID starting with 180b7629250c6670a21424e2b7be4e63d24d6b4d6e276b1118a7d616f1f2584a not found: ID does not exist" containerID="180b7629250c6670a21424e2b7be4e63d24d6b4d6e276b1118a7d616f1f2584a" Oct 31 14:04:51.854879 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.854836 5532 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"180b7629250c6670a21424e2b7be4e63d24d6b4d6e276b1118a7d616f1f2584a"} err="failed to get container status \"180b7629250c6670a21424e2b7be4e63d24d6b4d6e276b1118a7d616f1f2584a\": rpc error: code = NotFound desc = could not find container \"180b7629250c6670a21424e2b7be4e63d24d6b4d6e276b1118a7d616f1f2584a\": container with ID starting with 180b7629250c6670a21424e2b7be4e63d24d6b4d6e276b1118a7d616f1f2584a not found: ID does not exist" Oct 31 14:04:51.869185 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.869112 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg"] Oct 31 14:04:51.871276 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:51.871256 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-8b2f0a-758b55ccf9-kfzhg"] Oct 31 14:04:52.001541 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:52.001510 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e44283dd-c621-4479-bbd2-b392cfe2a176" path="/var/lib/kubelet/pods/e44283dd-c621-4479-bbd2-b392cfe2a176/volumes" Oct 31 14:04:54.362725 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.362690 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56949457b6-5s5g2"] Oct 31 14:04:54.363250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.363072 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e9d3729b-36e5-4ddd-ac81-ff4a7238254e" containerName="revoke-keys" Oct 31 14:04:54.363250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.363083 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9d3729b-36e5-4ddd-ac81-ff4a7238254e" containerName="revoke-keys" Oct 31 14:04:54.363250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.363104 5532 cpu_manager.go:414] "containerMap: removing container" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" Oct 31 14:04:54.363250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.363110 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" Oct 31 14:04:54.363250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.363118 5532 cpu_manager.go:414] "containerMap: removing container" podUID="e44283dd-c621-4479-bbd2-b392cfe2a176" containerName="payload-pre-processing" Oct 31 14:04:54.363250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.363124 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e44283dd-c621-4479-bbd2-b392cfe2a176" containerName="payload-pre-processing" Oct 31 14:04:54.363250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.363134 5532 cpu_manager.go:414] "containerMap: removing container" podUID="5d058e65-1d22-4861-9743-70b0ea3c2fad" containerName="revoke-keys" Oct 31 14:04:54.363250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.363139 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d058e65-1d22-4861-9743-70b0ea3c2fad" containerName="revoke-keys" Oct 31 14:04:54.363250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.363224 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d058e65-1d22-4861-9743-70b0ea3c2fad" containerName="revoke-keys" Oct 31 14:04:54.363250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.363237 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9d3729b-36e5-4ddd-ac81-ff4a7238254e" containerName="revoke-keys" Oct 31 14:04:54.363250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.363245 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="27dbfb95-a109-4e31-b1a0-4ac435c252d3" containerName="main" Oct 31 14:04:54.363250 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.363251 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="e44283dd-c621-4479-bbd2-b392cfe2a176" containerName="payload-pre-processing" Oct 31 14:04:54.366216 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.366196 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:04:54.372875 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.372851 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 14:04:54.373031 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.372853 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zjpdd\"" Oct 31 14:04:54.373031 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.372883 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 14:04:54.373291 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.372882 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 14:04:54.373291 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.372947 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 14:04:54.373291 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.372983 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 14:04:54.378680 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.378654 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56949457b6-5s5g2"] Oct 31 14:04:54.494562 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.494520 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-webhook-cert\") pod \"maas-controller-56949457b6-5s5g2\" (UID: \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\") " pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:04:54.494814 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.494676 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-metrics-certs\") pod \"maas-controller-56949457b6-5s5g2\" (UID: \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\") " pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:04:54.494905 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.494837 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg57l\" (UniqueName: \"kubernetes.io/projected/f0e0d9f0-3f6e-495e-9e50-01331ba34193-kube-api-access-hg57l\") pod \"maas-controller-56949457b6-5s5g2\" (UID: \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\") " pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:04:54.596027 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.595984 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg57l\" (UniqueName: \"kubernetes.io/projected/f0e0d9f0-3f6e-495e-9e50-01331ba34193-kube-api-access-hg57l\") pod \"maas-controller-56949457b6-5s5g2\" (UID: \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\") " pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:04:54.596253 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.596042 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-webhook-cert\") pod \"maas-controller-56949457b6-5s5g2\" (UID: \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\") " pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:04:54.596253 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.596080 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-metrics-certs\") pod \"maas-controller-56949457b6-5s5g2\" (UID: \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\") " pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:04:54.599260 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.599228 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-webhook-cert\") pod \"maas-controller-56949457b6-5s5g2\" (UID: \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\") " pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:04:54.599396 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.599228 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-metrics-certs\") pod \"maas-controller-56949457b6-5s5g2\" (UID: \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\") " pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:04:54.610890 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.610859 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg57l\" (UniqueName: \"kubernetes.io/projected/f0e0d9f0-3f6e-495e-9e50-01331ba34193-kube-api-access-hg57l\") pod \"maas-controller-56949457b6-5s5g2\" (UID: \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\") " pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:04:54.680821 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.680783 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:04:54.970106 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:54.970074 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56949457b6-5s5g2"] Oct 31 14:04:54.976745 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:04:54.976712 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0e0d9f0_3f6e_495e_9e50_01331ba34193.slice/crio-12864ca36e090d4f5c072a86c52b8744efc3de34bec487f7ccd2aa56ce20c4ce WatchSource:0}: Error finding container 12864ca36e090d4f5c072a86c52b8744efc3de34bec487f7ccd2aa56ce20c4ce: Status 404 returned error can't find the container with id 12864ca36e090d4f5c072a86c52b8744efc3de34bec487f7ccd2aa56ce20c4ce Oct 31 14:04:55.875127 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:55.875085 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56949457b6-5s5g2" event={"ID":"f0e0d9f0-3f6e-495e-9e50-01331ba34193","Type":"ContainerStarted","Data":"12864ca36e090d4f5c072a86c52b8744efc3de34bec487f7ccd2aa56ce20c4ce"} Oct 31 14:04:56.026630 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:56.026579 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs"] Oct 31 14:04:56.029792 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:56.029767 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33de6359-mk6fs"] Oct 31 14:04:56.882160 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:56.882061 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56949457b6-5s5g2" event={"ID":"f0e0d9f0-3f6e-495e-9e50-01331ba34193","Type":"ContainerStarted","Data":"7593a8cdc34ef6ed7cc92ce5f778b0be1fa31e0b257cb3c0255769118581f219"} Oct 31 14:04:56.882555 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:56.882160 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:04:56.906988 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:56.906935 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-56949457b6-5s5g2" podStartSLOduration=1.378296666 podStartE2EDuration="2.906920611s" totalImagesPullingTime="1.528623945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:04:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:04:56.90485589 +0000 UTC m=+3665.050599709" watchObservedRunningTime="2026-10-31 14:04:56.906920611 +0000 UTC m=+3665.052664447" Oct 31 14:04:56.982295 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:56.982264 5532 scope.go:122] "RemoveContainer" containerID="49df6c2c1b521f688f76d38704c202e8e98b31e4a3ec4837bbd99f511967645b" Oct 31 14:04:56.991338 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:56.991317 5532 scope.go:122] "RemoveContainer" containerID="5aaf419f866fba00ceda1fa4d11baac2628c546ffd7ab721af09b3001e56d54f" Oct 31 14:04:58.001980 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:04:58.001948 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01ce9637-0da3-4e30-af28-2966c675990f" path="/var/lib/kubelet/pods/01ce9637-0da3-4e30-af28-2966c675990f/volumes" Oct 31 14:05:01.760637 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:05:01.760583 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:05:07.893324 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:07.893289 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:05:13.815617 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.815565 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p"] Oct 31 14:05:13.819657 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.819631 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.836440 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.836407 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p"] Oct 31 14:05:13.838106 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.838076 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5"] Oct 31 14:05:13.842076 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.842049 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.860233 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.860198 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5"] Oct 31 14:05:13.879985 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.879955 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.879985 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.879987 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.880194 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880009 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5stg9\" (UniqueName: \"kubernetes.io/projected/7d461f31-7477-4c4f-81fa-6a4769e2266f-kube-api-access-5stg9\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.880194 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880093 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.880194 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880136 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpvnp\" (UniqueName: \"kubernetes.io/projected/532959bd-96e4-40f7-a157-7302d6216a41-kube-api-access-dpvnp\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.880194 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880156 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.880194 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880188 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.880433 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880238 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d461f31-7477-4c4f-81fa-6a4769e2266f-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.880433 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880292 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532959bd-96e4-40f7-a157-7302d6216a41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.880433 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880324 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.880433 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880415 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.880688 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880462 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.880688 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880502 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532959bd-96e4-40f7-a157-7302d6216a41-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.880688 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880533 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.880688 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880570 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532959bd-96e4-40f7-a157-7302d6216a41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.880688 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880620 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.880688 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880644 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d461f31-7477-4c4f-81fa-6a4769e2266f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.880688 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.880662 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d461f31-7477-4c4f-81fa-6a4769e2266f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.981583 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.981536 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.981583 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.981587 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d461f31-7477-4c4f-81fa-6a4769e2266f-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.981870 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.981696 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532959bd-96e4-40f7-a157-7302d6216a41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.981870 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.981721 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.981870 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.981754 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.981870 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.981780 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.981870 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.981804 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532959bd-96e4-40f7-a157-7302d6216a41-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.981870 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.981833 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.982188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.981875 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532959bd-96e4-40f7-a157-7302d6216a41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.982188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.981904 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.982188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.981935 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d461f31-7477-4c4f-81fa-6a4769e2266f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.982188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.981965 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d461f31-7477-4c4f-81fa-6a4769e2266f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.982188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.981998 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.982188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.982022 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.982188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.982031 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.982188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.982057 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5stg9\" (UniqueName: \"kubernetes.io/projected/7d461f31-7477-4c4f-81fa-6a4769e2266f-kube-api-access-5stg9\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.982188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.982112 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.982188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.982171 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.982188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.982177 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpvnp\" (UniqueName: \"kubernetes.io/projected/532959bd-96e4-40f7-a157-7302d6216a41-kube-api-access-dpvnp\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.983227 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.982215 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.983227 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.982234 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.983227 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.982334 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.983227 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.982459 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.983227 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.982513 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/532959bd-96e4-40f7-a157-7302d6216a41-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.983227 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.982839 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.983227 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.982861 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.983227 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.983000 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.983633 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.983439 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7d461f31-7477-4c4f-81fa-6a4769e2266f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.984715 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.984691 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/532959bd-96e4-40f7-a157-7302d6216a41-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.984923 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.984906 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7d461f31-7477-4c4f-81fa-6a4769e2266f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.985068 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.985048 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7d461f31-7477-4c4f-81fa-6a4769e2266f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.985239 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.985222 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/532959bd-96e4-40f7-a157-7302d6216a41-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.996221 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.996197 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpvnp\" (UniqueName: \"kubernetes.io/projected/532959bd-96e4-40f7-a157-7302d6216a41-kube-api-access-dpvnp\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.996221 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.996219 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7d461f31-7477-4c4f-81fa-6a4769e2266f-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:13.996434 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.996254 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/532959bd-96e4-40f7-a157-7302d6216a41-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-tvm4p\" (UID: \"532959bd-96e4-40f7-a157-7302d6216a41\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:13.996756 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:13.996735 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5stg9\" (UniqueName: \"kubernetes.io/projected/7d461f31-7477-4c4f-81fa-6a4769e2266f-kube-api-access-5stg9\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-76bb5\" (UID: \"7d461f31-7477-4c4f-81fa-6a4769e2266f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:14.135735 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:14.135644 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:14.162559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:14.162492 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:14.425358 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:14.425192 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p"] Oct 31 14:05:14.428254 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:05:14.428222 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod532959bd_96e4_40f7_a157_7302d6216a41.slice/crio-ca3d6e3698689b2dba4e866d6f238ded5b28ed0950aea024b94c47e572ba2816 WatchSource:0}: Error finding container ca3d6e3698689b2dba4e866d6f238ded5b28ed0950aea024b94c47e572ba2816: Status 404 returned error can't find the container with id ca3d6e3698689b2dba4e866d6f238ded5b28ed0950aea024b94c47e572ba2816 Oct 31 14:05:14.430855 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:14.430821 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:05:14.430973 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:14.430929 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:05:14.431042 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:14.430974 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:05:14.453987 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:14.453960 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5"] Oct 31 14:05:14.462901 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:05:14.462874 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d461f31_7477_4c4f_81fa_6a4769e2266f.slice/crio-064355d127c546816d373e56baaad168889ba1c692ec0d26e1e80264abbdc504 WatchSource:0}: Error finding container 064355d127c546816d373e56baaad168889ba1c692ec0d26e1e80264abbdc504: Status 404 returned error can't find the container with id 064355d127c546816d373e56baaad168889ba1c692ec0d26e1e80264abbdc504 Oct 31 14:05:14.465765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:14.465725 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:05:14.465867 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:14.465818 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:05:14.465912 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:14.465863 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:05:15.001290 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:15.001254 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" event={"ID":"7d461f31-7477-4c4f-81fa-6a4769e2266f","Type":"ContainerStarted","Data":"1632f480328949846c65224e0e042ddd5bd52ede00e08cd7362e18c580d990e5"} Oct 31 14:05:15.001806 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:15.001775 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" event={"ID":"7d461f31-7477-4c4f-81fa-6a4769e2266f","Type":"ContainerStarted","Data":"064355d127c546816d373e56baaad168889ba1c692ec0d26e1e80264abbdc504"} Oct 31 14:05:15.003267 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:15.003241 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" event={"ID":"532959bd-96e4-40f7-a157-7302d6216a41","Type":"ContainerStarted","Data":"65017f61e0b2ab4461fa901644fe46a1634e8d304b7b14ab8dbb6466b92a0f46"} Oct 31 14:05:15.003465 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:15.003285 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" event={"ID":"532959bd-96e4-40f7-a157-7302d6216a41","Type":"ContainerStarted","Data":"ca3d6e3698689b2dba4e866d6f238ded5b28ed0950aea024b94c47e572ba2816"} Oct 31 14:05:15.030325 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:15.030273 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" podStartSLOduration=2.030252037 podStartE2EDuration="2.030252037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:05:15.027914604 +0000 UTC m=+3683.173658429" watchObservedRunningTime="2026-10-31 14:05:15.030252037 +0000 UTC m=+3683.175995864" Oct 31 14:05:15.052643 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:15.052556 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" podStartSLOduration=2.052534882 podStartE2EDuration="2.052534882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:05:15.05167846 +0000 UTC m=+3683.197422289" watchObservedRunningTime="2026-10-31 14:05:15.052534882 +0000 UTC m=+3683.198278709" Oct 31 14:05:15.135932 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:15.135901 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:15.137574 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:15.137546 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 31 14:05:15.137726 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:15.137608 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" podUID="532959bd-96e4-40f7-a157-7302d6216a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 31 14:05:15.163620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:15.162996 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:15.180617 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:15.171878 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-76bb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Oct 31 14:05:15.180617 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:15.171945 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" podUID="7d461f31-7477-4c4f-81fa-6a4769e2266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Oct 31 14:05:16.136581 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:16.136547 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 31 14:05:16.137083 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:16.136621 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" podUID="532959bd-96e4-40f7-a157-7302d6216a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 31 14:05:16.169620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:16.168133 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-76bb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Oct 31 14:05:16.169620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:16.168212 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" podUID="7d461f31-7477-4c4f-81fa-6a4769e2266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Oct 31 14:05:17.136203 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:17.136162 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 31 14:05:17.136388 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:17.136234 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" podUID="532959bd-96e4-40f7-a157-7302d6216a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 31 14:05:17.163426 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:17.163000 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-76bb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Oct 31 14:05:17.163426 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:17.163062 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" podUID="7d461f31-7477-4c4f-81fa-6a4769e2266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Oct 31 14:05:18.138656 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:18.136291 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 31 14:05:18.138656 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:18.136342 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" podUID="532959bd-96e4-40f7-a157-7302d6216a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 31 14:05:18.170854 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:18.170802 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-76bb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Oct 31 14:05:18.171498 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:18.171446 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" podUID="7d461f31-7477-4c4f-81fa-6a4769e2266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Oct 31 14:05:19.136485 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:19.136449 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 31 14:05:19.136679 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:19.136504 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" podUID="532959bd-96e4-40f7-a157-7302d6216a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 31 14:05:19.167630 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:19.165156 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-76bb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Oct 31 14:05:19.167630 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:19.165216 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" podUID="7d461f31-7477-4c4f-81fa-6a4769e2266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Oct 31 14:05:20.136572 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:20.136532 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 31 14:05:20.137012 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:20.136622 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" podUID="532959bd-96e4-40f7-a157-7302d6216a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 31 14:05:20.167758 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:20.167719 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-76bb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Oct 31 14:05:20.168033 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:20.168006 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" podUID="7d461f31-7477-4c4f-81fa-6a4769e2266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Oct 31 14:05:21.136587 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:21.136551 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" start-of-body= Oct 31 14:05:21.137013 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:21.136624 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" podUID="532959bd-96e4-40f7-a157-7302d6216a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": dial tcp 10.132.1.78:15021: connect: connection refused" Oct 31 14:05:21.163634 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:21.163580 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-76bb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" start-of-body= Oct 31 14:05:21.163800 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:21.163666 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" podUID="7d461f31-7477-4c4f-81fa-6a4769e2266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": dial tcp 10.132.1.79:15021: connect: connection refused" Oct 31 14:05:23.137344 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:23.137295 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:05:23.137896 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:23.137379 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" podUID="532959bd-96e4-40f7-a157-7302d6216a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:05:23.164194 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:23.164160 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-76bb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:05:23.164374 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:23.164213 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" podUID="7d461f31-7477-4c4f-81fa-6a4769e2266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:05:24.178803 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:24.178761 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:05:24.179481 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:24.179443 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" podUID="532959bd-96e4-40f7-a157-7302d6216a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:05:24.206805 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:24.206767 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-76bb5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:05:24.206965 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:24.206828 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" podUID="7d461f31-7477-4c4f-81fa-6a4769e2266f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:05:25.180461 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:25.180416 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:05:25.181082 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:25.181041 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" podUID="532959bd-96e4-40f7-a157-7302d6216a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:05:25.203559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:25.203527 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:25.203952 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:25.203922 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:25.204721 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:25.204696 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-76bb5" Oct 31 14:05:25.323311 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:25.322649 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:25.323311 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:25.323128 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:25.323848 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:25.323824 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" Oct 31 14:05:25.844692 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:25.844650 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56949457b6-5s5g2"] Oct 31 14:05:25.844966 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:25.844905 5532 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-56949457b6-5s5g2" podUID="f0e0d9f0-3f6e-495e-9e50-01331ba34193" containerName="manager" containerID="cri-o://7593a8cdc34ef6ed7cc92ce5f778b0be1fa31e0b257cb3c0255769118581f219" gracePeriod=30 Oct 31 14:05:26.071840 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:26.071809 5532 generic.go:356] "Generic (PLEG): container finished" podID="f0e0d9f0-3f6e-495e-9e50-01331ba34193" containerID="7593a8cdc34ef6ed7cc92ce5f778b0be1fa31e0b257cb3c0255769118581f219" exitCode=0 Oct 31 14:05:26.072018 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:26.071886 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56949457b6-5s5g2" event={"ID":"f0e0d9f0-3f6e-495e-9e50-01331ba34193","Type":"ContainerDied","Data":"7593a8cdc34ef6ed7cc92ce5f778b0be1fa31e0b257cb3c0255769118581f219"} Oct 31 14:05:26.103329 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:26.103276 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:05:26.227675 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:26.227631 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0e0d9f0-3f6e-495e-9e50-01331ba34193-kube-api-access-hg57l\" (UniqueName: \"kubernetes.io/projected/f0e0d9f0-3f6e-495e-9e50-01331ba34193-kube-api-access-hg57l\") pod \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\" (UID: \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\") " Oct 31 14:05:26.228158 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:26.227755 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-metrics-certs\") pod \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\" (UID: \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\") " Oct 31 14:05:26.228158 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:26.227794 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-webhook-cert\") pod \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\" (UID: \"f0e0d9f0-3f6e-495e-9e50-01331ba34193\") " Oct 31 14:05:26.230029 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:26.229991 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0e0d9f0-3f6e-495e-9e50-01331ba34193-kube-api-access-hg57l" pod "f0e0d9f0-3f6e-495e-9e50-01331ba34193" (UID: "f0e0d9f0-3f6e-495e-9e50-01331ba34193"). InnerVolumeSpecName "kube-api-access-hg57l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:05:26.230165 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:26.230093 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-metrics-certs" pod "f0e0d9f0-3f6e-495e-9e50-01331ba34193" (UID: "f0e0d9f0-3f6e-495e-9e50-01331ba34193"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:05:26.230165 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:26.230117 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-webhook-cert" pod "f0e0d9f0-3f6e-495e-9e50-01331ba34193" (UID: "f0e0d9f0-3f6e-495e-9e50-01331ba34193"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:05:26.329223 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:26.329186 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg57l\" (UniqueName: \"kubernetes.io/projected/f0e0d9f0-3f6e-495e-9e50-01331ba34193-kube-api-access-hg57l\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:26.329223 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:26.329218 5532 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-metrics-certs\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:26.329223 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:26.329229 5532 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f0e0d9f0-3f6e-495e-9e50-01331ba34193-webhook-cert\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:05:27.079138 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:27.079109 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56949457b6-5s5g2" Oct 31 14:05:27.079138 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:27.079130 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56949457b6-5s5g2" event={"ID":"f0e0d9f0-3f6e-495e-9e50-01331ba34193","Type":"ContainerDied","Data":"12864ca36e090d4f5c072a86c52b8744efc3de34bec487f7ccd2aa56ce20c4ce"} Oct 31 14:05:27.079400 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:27.079215 5532 scope.go:122] "RemoveContainer" containerID="7593a8cdc34ef6ed7cc92ce5f778b0be1fa31e0b257cb3c0255769118581f219" Oct 31 14:05:27.106097 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:27.106068 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-56949457b6-5s5g2"] Oct 31 14:05:27.113291 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:27.113266 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-56949457b6-5s5g2"] Oct 31 14:05:28.002636 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.002605 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0e0d9f0-3f6e-495e-9e50-01331ba34193" path="/var/lib/kubelet/pods/f0e0d9f0-3f6e-495e-9e50-01331ba34193/volumes" Oct 31 14:05:28.342635 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.342540 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-56949457b6-sffrf"] Oct 31 14:05:28.343001 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.342987 5532 cpu_manager.go:414] "containerMap: removing container" podUID="f0e0d9f0-3f6e-495e-9e50-01331ba34193" containerName="manager" Oct 31 14:05:28.343047 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.343002 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0e0d9f0-3f6e-495e-9e50-01331ba34193" containerName="manager" Oct 31 14:05:28.343106 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.343095 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0e0d9f0-3f6e-495e-9e50-01331ba34193" containerName="manager" Oct 31 14:05:28.348988 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.348969 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56949457b6-sffrf" Oct 31 14:05:28.358008 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.357984 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56949457b6-sffrf"] Oct 31 14:05:28.360772 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.360747 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 14:05:28.360904 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.360828 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 31 14:05:28.361437 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.361421 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 31 14:05:28.361635 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.361615 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 14:05:28.361735 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.361721 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zjpdd\"" Oct 31 14:05:28.361788 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.361728 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 14:05:28.451093 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.451053 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0-webhook-cert\") pod \"maas-controller-56949457b6-sffrf\" (UID: \"1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0\") " pod="opendatahub/maas-controller-56949457b6-sffrf" Oct 31 14:05:28.451314 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.451118 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0-metrics-certs\") pod \"maas-controller-56949457b6-sffrf\" (UID: \"1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0\") " pod="opendatahub/maas-controller-56949457b6-sffrf" Oct 31 14:05:28.451314 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.451277 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvjps\" (UniqueName: \"kubernetes.io/projected/1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0-kube-api-access-hvjps\") pod \"maas-controller-56949457b6-sffrf\" (UID: \"1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0\") " pod="opendatahub/maas-controller-56949457b6-sffrf" Oct 31 14:05:28.552118 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.552084 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0-metrics-certs\") pod \"maas-controller-56949457b6-sffrf\" (UID: \"1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0\") " pod="opendatahub/maas-controller-56949457b6-sffrf" Oct 31 14:05:28.552327 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.552156 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvjps\" (UniqueName: \"kubernetes.io/projected/1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0-kube-api-access-hvjps\") pod \"maas-controller-56949457b6-sffrf\" (UID: \"1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0\") " pod="opendatahub/maas-controller-56949457b6-sffrf" Oct 31 14:05:28.552327 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.552208 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0-webhook-cert\") pod \"maas-controller-56949457b6-sffrf\" (UID: \"1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0\") " pod="opendatahub/maas-controller-56949457b6-sffrf" Oct 31 14:05:28.554683 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.554657 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0-webhook-cert\") pod \"maas-controller-56949457b6-sffrf\" (UID: \"1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0\") " pod="opendatahub/maas-controller-56949457b6-sffrf" Oct 31 14:05:28.554802 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.554695 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0-metrics-certs\") pod \"maas-controller-56949457b6-sffrf\" (UID: \"1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0\") " pod="opendatahub/maas-controller-56949457b6-sffrf" Oct 31 14:05:28.569097 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.569077 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvjps\" (UniqueName: \"kubernetes.io/projected/1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0-kube-api-access-hvjps\") pod \"maas-controller-56949457b6-sffrf\" (UID: \"1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0\") " pod="opendatahub/maas-controller-56949457b6-sffrf" Oct 31 14:05:28.664904 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.664868 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-56949457b6-sffrf" Oct 31 14:05:28.864230 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.864194 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4"] Oct 31 14:05:28.869161 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.869133 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:28.884813 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.884785 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4"] Oct 31 14:05:28.943887 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.943864 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-56949457b6-sffrf"] Oct 31 14:05:28.946156 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:05:28.946122 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8ac6e8_9c7c_4370_a5a3_0ca659c44da0.slice/crio-32a2eaf8df2c873be2d61b1e449fbfaeaed1ae8c3a85610a48e5b2f95416ee94 WatchSource:0}: Error finding container 32a2eaf8df2c873be2d61b1e449fbfaeaed1ae8c3a85610a48e5b2f95416ee94: Status 404 returned error can't find the container with id 32a2eaf8df2c873be2d61b1e449fbfaeaed1ae8c3a85610a48e5b2f95416ee94 Oct 31 14:05:28.956562 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.956537 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:28.956669 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.956570 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:28.956669 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.956589 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:28.956746 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.956717 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:28.956784 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.956749 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:28.956784 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.956768 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:28.956850 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.956825 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:28.956904 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.956886 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:28.956944 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:28.956914 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phl6f\" (UniqueName: \"kubernetes.io/projected/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-kube-api-access-phl6f\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.026498 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.026472 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z"] Oct 31 14:05:29.030366 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.030341 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c452f5d4-6jh2z"] Oct 31 14:05:29.058106 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.058078 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.058212 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.058110 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.058212 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.058133 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.058322 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.058207 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.058322 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.058266 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.058322 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.058297 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.058483 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.058403 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.058483 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.058476 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-workload-certs\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.058626 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.058542 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-workload-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.058626 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.058571 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.058746 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.058634 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phl6f\" (UniqueName: \"kubernetes.io/projected/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-kube-api-access-phl6f\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.058746 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.058682 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istio-data\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.058892 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.058853 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-credential-socket\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.059131 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.059111 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.060503 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.060483 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istio-envoy\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.060792 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.060773 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.079294 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.079263 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-istio-token\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.079402 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.079350 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phl6f\" (UniqueName: \"kubernetes.io/projected/fdf3bdf7-1bcc-43e5-bccd-a375cce224fd-kube-api-access-phl6f\") pod \"maas-default-gateway-openshift-default-5d84bc6b95-qhgv4\" (UID: \"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.093317 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.093292 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56949457b6-sffrf" event={"ID":"1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0","Type":"ContainerStarted","Data":"32a2eaf8df2c873be2d61b1e449fbfaeaed1ae8c3a85610a48e5b2f95416ee94"} Oct 31 14:05:29.184194 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.184160 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:29.469943 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.469914 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4"] Oct 31 14:05:29.471755 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:05:29.471724 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdf3bdf7_1bcc_43e5_bccd_a375cce224fd.slice/crio-878a5f73c55513cb76f78c395b18a5862ea06e0450512ef9863745c78fb0e4c9 WatchSource:0}: Error finding container 878a5f73c55513cb76f78c395b18a5862ea06e0450512ef9863745c78fb0e4c9: Status 404 returned error can't find the container with id 878a5f73c55513cb76f78c395b18a5862ea06e0450512ef9863745c78fb0e4c9 Oct 31 14:05:29.474280 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.474240 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:05:29.474403 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.474314 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:05:29.474403 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:29.474343 5532 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"} Oct 31 14:05:30.001685 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:30.001654 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ae2bee7-37b1-4b54-8c60-3256a06e478a" path="/var/lib/kubelet/pods/7ae2bee7-37b1-4b54-8c60-3256a06e478a/volumes" Oct 31 14:05:30.102044 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:30.102007 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" event={"ID":"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd","Type":"ContainerStarted","Data":"bdc048ae0f8bad0ce7181f980050924656c3b7f4997e358e788f832516067df3"} Oct 31 14:05:30.102559 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:30.102070 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" event={"ID":"fdf3bdf7-1bcc-43e5-bccd-a375cce224fd","Type":"ContainerStarted","Data":"878a5f73c55513cb76f78c395b18a5862ea06e0450512ef9863745c78fb0e4c9"} Oct 31 14:05:30.103964 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:30.103934 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-56949457b6-sffrf" event={"ID":"1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0","Type":"ContainerStarted","Data":"86412c27e29b7607e83a2e84ef0c16373e79ce887ccbd8f3d8a4e8ae835bedf7"} Oct 31 14:05:30.104084 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:30.103985 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-56949457b6-sffrf" Oct 31 14:05:30.137747 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:30.137701 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" podStartSLOduration=2.137688642 podStartE2EDuration="2.137688642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:05:30.135719665 +0000 UTC m=+3698.281463493" watchObservedRunningTime="2026-10-31 14:05:30.137688642 +0000 UTC m=+3698.283432469" Oct 31 14:05:30.174766 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:30.174714 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-56949457b6-sffrf" podStartSLOduration=1.738298673 podStartE2EDuration="2.174699459s" totalImagesPullingTime="436.400786ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:05:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:05:30.173648373 +0000 UTC m=+3698.319392201" watchObservedRunningTime="2026-10-31 14:05:30.174699459 +0000 UTC m=+3698.320443287" Oct 31 14:05:30.184774 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:30.184746 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:30.186023 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:30.185999 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 14:05:30.186125 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:30.186041 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" podUID="fdf3bdf7-1bcc-43e5-bccd-a375cce224fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 14:05:31.190316 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:31.190271 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 14:05:31.190740 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:31.190339 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" podUID="fdf3bdf7-1bcc-43e5-bccd-a375cce224fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 14:05:31.783720 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:05:31.783684 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:05:32.185472 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:32.185428 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 14:05:32.185703 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:32.185490 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" podUID="fdf3bdf7-1bcc-43e5-bccd-a375cce224fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 14:05:33.185276 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:33.185241 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 14:05:33.185708 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:33.185292 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" podUID="fdf3bdf7-1bcc-43e5-bccd-a375cce224fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 14:05:34.184659 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:34.184615 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 14:05:34.184864 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:34.184676 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" podUID="fdf3bdf7-1bcc-43e5-bccd-a375cce224fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 14:05:35.185319 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:35.185282 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 14:05:35.185798 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:35.185336 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" podUID="fdf3bdf7-1bcc-43e5-bccd-a375cce224fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 14:05:36.185092 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:36.185056 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" start-of-body= Oct 31 14:05:36.185305 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:36.185111 5532 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" podUID="fdf3bdf7-1bcc-43e5-bccd-a375cce224fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": dial tcp 10.132.1.81:15021: connect: connection refused" Oct 31 14:05:37.666056 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:37.666021 5532 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:37.666546 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:37.666420 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:37.667611 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:37.667568 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" Oct 31 14:05:39.029632 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:39.029577 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv"] Oct 31 14:05:39.032533 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:39.032506 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ab0e8c0d-p22qv"] Oct 31 14:05:40.001842 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:40.001805 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1def377-c1f7-475d-a344-1e03e49b0fc4" path="/var/lib/kubelet/pods/c1def377-c1f7-475d-a344-1e03e49b0fc4/volumes" Oct 31 14:05:41.112327 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:41.112299 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-56949457b6-sffrf" Oct 31 14:05:43.026013 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:43.025974 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q"] Oct 31 14:05:43.029642 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:43.029612 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2fee3c04-c984q"] Oct 31 14:05:44.001575 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:44.001543 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a006a12c-ca86-45e0-8b7b-1735b3d02dd4" path="/var/lib/kubelet/pods/a006a12c-ca86-45e0-8b7b-1735b3d02dd4/volumes" Oct 31 14:05:57.049441 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:57.049407 5532 scope.go:122] "RemoveContainer" containerID="022b0a140dd2a317409a5beda16203b02bfd9efdb9d65abc4338db1cce2c359d" Oct 31 14:05:57.062120 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:57.062097 5532 scope.go:122] "RemoveContainer" containerID="dffa9693f69f07049995d1663e40ac14cf4c3d8c689c5104c7889d79302dcc6f" Oct 31 14:05:57.070656 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:57.070635 5532 scope.go:122] "RemoveContainer" containerID="0f93e3640e20d465d7658b19eec7844408b7b5dc01901708deb0fe93d6fd397f" Oct 31 14:05:57.078662 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:57.078645 5532 scope.go:122] "RemoveContainer" containerID="ba6514182c916cb73079ac6b89aaeb6e2e96d2616073c9d8fdc70de71408369a" Oct 31 14:05:57.086868 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:57.086853 5532 scope.go:122] "RemoveContainer" containerID="ef165bec02e4019d523463ef009c1a683a7f3e35542aaa22d926721b58bf855b" Oct 31 14:05:57.094548 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:05:57.094533 5532 scope.go:122] "RemoveContainer" containerID="faecc4f125de25fe04ad568f2c48a46c59487aa569b39ff2faa4463714763bd3" Oct 31 14:06:01.011060 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:06:01.011029 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:06:01.774536 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:06:01.774497 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:06:16.868518 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:06:16.868479 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:06:21.007607 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:06:21.007555 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:06:31.761421 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:06:31.761381 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:06:37.903357 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:06:37.903325 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:06:50.804211 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:06:50.804174 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:06:53.725768 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:06:53.725726 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:06:53.726492 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:06:53.726448 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" podUID="fdf3bdf7-1bcc-43e5-bccd-a375cce224fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:06:57.139283 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:06:57.139250 5532 scope.go:122] "RemoveContainer" containerID="1e9932aebf1f8fb643883c7fd1526f2e637be1e6094aa362974b7f294245b762" Oct 31 14:06:57.148129 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:06:57.148112 5532 scope.go:122] "RemoveContainer" containerID="4fca01f9392b987f37d79662a2cb58ef131895256ab23a0b43075c8570456c60" Oct 31 14:06:57.155916 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:06:57.155898 5532 scope.go:122] "RemoveContainer" containerID="6d708f50d1b6273986aed005b574290159f2cbfa43e29ed3d78a50c52264660c" Oct 31 14:06:57.163674 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:06:57.163656 5532 scope.go:122] "RemoveContainer" containerID="c05d887bb660afca7722a251f59d0c9a1e05633cacacb52be42fcc43e1168136" Oct 31 14:06:57.173286 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:06:57.173270 5532 scope.go:122] "RemoveContainer" containerID="1b7ee7729977cd8e6bf4071acba25268554e73b5dcd79ca3f015580846144cdd" Oct 31 14:07:01.760645 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:07:01.760546 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:07:15.314089 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:15.314048 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:07:25.031728 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:25.031693 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5"] Oct 31 14:07:25.045166 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:25.036945 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e6652ad-qj9f5"] Oct 31 14:07:25.503414 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:25.503385 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:07:26.002272 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:26.002209 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed22fd13-4e70-4c82-9b23-be39ca96e9e5" path="/var/lib/kubelet/pods/ed22fd13-4e70-4c82-9b23-be39ca96e9e5/volumes" Oct 31 14:07:29.535620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:29.535405 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:07:31.760394 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:07:31.760354 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:07:46.392808 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:46.392772 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:07:50.435634 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:50.434725 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:07:52.031224 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:52.031194 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn"] Oct 31 14:07:52.034776 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:52.034748 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1cba2c3f-sdtcn"] Oct 31 14:07:54.001572 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:54.001537 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a17552fd-3d5f-4e30-8f8a-90780d4c47b2" path="/var/lib/kubelet/pods/a17552fd-3d5f-4e30-8f8a-90780d4c47b2/volumes" Oct 31 14:07:57.208068 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:57.208034 5532 scope.go:122] "RemoveContainer" containerID="95b832975fb32221bbb5f8bac0f874376d2307bf7540e8fd3da5a9452fcef12b" Oct 31 14:07:57.217094 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:57.217073 5532 scope.go:122] "RemoveContainer" containerID="7d49a3590be14a5ae0843886ce1daf5f8213e37a98bccf8c6143ea7470c0bf26" Oct 31 14:07:57.225130 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:57.225108 5532 scope.go:122] "RemoveContainer" containerID="809ee30d98bb3565ba5cf155951221a5005129295bb0d9bd140d1bb6d4d0f599" Oct 31 14:07:57.233172 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:07:57.233152 5532 scope.go:122] "RemoveContainer" containerID="1fdd33f46a7de9f801a9e43fada789f6bfabeeb20f78223bd125b38412f05d63" Oct 31 14:08:01.761081 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:08:01.761043 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:08:04.628414 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:08:04.628375 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:08:10.290619 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:08:10.289787 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:08:23.513103 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:08:23.513063 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:08:27.436606 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:08:27.436536 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:08:31.762232 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:08:31.762143 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:08:41.513274 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:08:41.513233 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:08:52.111845 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:08:52.111808 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:08:56.855620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:08:56.854654 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:08:57.273241 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:08:57.273181 5532 scope.go:122] "RemoveContainer" containerID="5103df28eb2bb9f461842120748baaff2767a3f550bd46ac2f1f46244a87fdf5" Oct 31 14:08:57.330618 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:08:57.324659 5532 scope.go:122] "RemoveContainer" containerID="dca19a4423cc4dd258cccd32bbd588c5b152623a7ef4389b248953d5919280ab" Oct 31 14:08:57.374014 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:08:57.373878 5532 scope.go:122] "RemoveContainer" containerID="6297d21deadfd766a7a79645680530c0a03042acabc06ba77b69aa07df4ee2be" Oct 31 14:09:01.760124 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:09:01.760083 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:09:20.607156 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:20.607121 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:09:23.714215 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:23.713708 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:09:23.714215 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:23.713777 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-qhgv4" podUID="fdf3bdf7-1bcc-43e5-bccd-a375cce224fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:09:30.033729 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:30.033690 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f"] Oct 31 14:09:30.035128 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:30.035102 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb"] Oct 31 14:09:30.037043 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:30.037017 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91f2bda2-bqt9f"] Oct 31 14:09:30.040209 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:30.040184 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed40470d-vjtsb"] Oct 31 14:09:31.760402 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:09:31.760367 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:09:32.002481 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:32.002447 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d058e65-1d22-4861-9743-70b0ea3c2fad" path="/var/lib/kubelet/pods/5d058e65-1d22-4861-9743-70b0ea3c2fad/volumes" Oct 31 14:09:32.002902 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:32.002887 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9d3729b-36e5-4ddd-ac81-ff4a7238254e" path="/var/lib/kubelet/pods/e9d3729b-36e5-4ddd-ac81-ff4a7238254e/volumes" Oct 31 14:09:38.992144 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:38.992098 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:09:41.369957 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:41.369914 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:09:41.370656 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:41.370619 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-tvm4p" podUID="532959bd-96e4-40f7-a157-7302d6216a41" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:09:49.291220 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:49.291185 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:09:57.576408 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:09:57.576377 5532 scope.go:122] "RemoveContainer" containerID="e1ce1be704fc4bedaab993f01d6fb96fcfa6e631175c1a053d68c3a6c70daf0f" Oct 31 14:10:01.760875 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:10:01.760791 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:10:05.600401 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:10:05.600365 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:10:10.280295 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:10:10.280257 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:10:23.531844 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:10:23.531806 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:10:31.761071 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:10:31.761031 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:10:38.423844 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:10:38.423805 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:10:54.216848 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:10:54.216812 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:10:57.600015 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:10:57.599968 5532 scope.go:122] "RemoveContainer" containerID="4cc6167f4a52acb180ff28137da87dd5d8d1b31d86faa1444bf77dae7046fa90" Oct 31 14:10:57.608496 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:10:57.608475 5532 scope.go:122] "RemoveContainer" containerID="66c332cb521624b5fefad7a28ac9b6307c516655dd5694443c0ded237a3cb043" Oct 31 14:10:57.616422 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:10:57.616402 5532 scope.go:122] "RemoveContainer" containerID="2b1f3fa31c8593da77fd7fcb1ee8201210d8f54333677fbb0bb7ebd1311e8aad" Oct 31 14:11:01.761564 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:11:01.761522 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:11:06.396769 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.396729 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n"] Oct 31 14:11:06.402488 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.402467 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" Oct 31 14:11:06.410035 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.409847 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:11:06.410035 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.409913 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:11:06.410035 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.410003 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:11:06.411399 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.411106 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n"] Oct 31 14:11:06.411399 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.411302 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:11:06.478200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.478163 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbpvl\" (UniqueName: \"kubernetes.io/projected/c0c76746-d068-475c-9ac5-856f863dac4a-kube-api-access-lbpvl\") pod \"maas-api-revoke-sub-cd0c3124-42v4n\" (UID: \"c0c76746-d068-475c-9ac5-856f863dac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" Oct 31 14:11:06.478379 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.478248 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0c3124-42v4n\" (UID: \"c0c76746-d068-475c-9ac5-856f863dac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" Oct 31 14:11:06.478427 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.478380 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0c3124-42v4n\" (UID: \"c0c76746-d068-475c-9ac5-856f863dac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" Oct 31 14:11:06.579842 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.579805 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0c3124-42v4n\" (UID: \"c0c76746-d068-475c-9ac5-856f863dac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" Oct 31 14:11:06.580021 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.579949 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0c3124-42v4n\" (UID: \"c0c76746-d068-475c-9ac5-856f863dac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" Oct 31 14:11:06.580021 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.579997 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbpvl\" (UniqueName: \"kubernetes.io/projected/c0c76746-d068-475c-9ac5-856f863dac4a-kube-api-access-lbpvl\") pod \"maas-api-revoke-sub-cd0c3124-42v4n\" (UID: \"c0c76746-d068-475c-9ac5-856f863dac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" Oct 31 14:11:06.580474 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.580452 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd0c3124-42v4n\" (UID: \"c0c76746-d068-475c-9ac5-856f863dac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" Oct 31 14:11:06.582463 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.582442 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd0c3124-42v4n\" (UID: \"c0c76746-d068-475c-9ac5-856f863dac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" Oct 31 14:11:06.595281 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.595254 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbpvl\" (UniqueName: \"kubernetes.io/projected/c0c76746-d068-475c-9ac5-856f863dac4a-kube-api-access-lbpvl\") pod \"maas-api-revoke-sub-cd0c3124-42v4n\" (UID: \"c0c76746-d068-475c-9ac5-856f863dac4a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" Oct 31 14:11:06.717200 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:06.717119 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" Oct 31 14:11:07.000830 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:07.000802 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n"] Oct 31 14:11:07.003161 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:11:07.003129 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c76746_d068_475c_9ac5_856f863dac4a.slice/crio-152e57f94ae75ba768ca1724d69ae12da63b248e1ced25dd62c76d300bc42d73 WatchSource:0}: Error finding container 152e57f94ae75ba768ca1724d69ae12da63b248e1ced25dd62c76d300bc42d73: Status 404 returned error can't find the container with id 152e57f94ae75ba768ca1724d69ae12da63b248e1ced25dd62c76d300bc42d73 Oct 31 14:11:07.371871 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:07.371829 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" event={"ID":"c0c76746-d068-475c-9ac5-856f863dac4a","Type":"ContainerStarted","Data":"0cf83325a6e8fdbfa60eacf33f9863439501d5b8d9c89e5a5a27ac5a425faf6b"} Oct 31 14:11:07.372055 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:07.371890 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" event={"ID":"c0c76746-d068-475c-9ac5-856f863dac4a","Type":"ContainerStarted","Data":"152e57f94ae75ba768ca1724d69ae12da63b248e1ced25dd62c76d300bc42d73"} Oct 31 14:11:07.393126 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:07.393080 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" podStartSLOduration=1.3930657339999999 podStartE2EDuration="1.393065734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:11:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:11:07.39210916 +0000 UTC m=+4035.537852987" watchObservedRunningTime="2026-10-31 14:11:07.393065734 +0000 UTC m=+4035.538809562" Oct 31 14:11:08.378658 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:08.378620 5532 generic.go:356] "Generic (PLEG): container finished" podID="c0c76746-d068-475c-9ac5-856f863dac4a" containerID="0cf83325a6e8fdbfa60eacf33f9863439501d5b8d9c89e5a5a27ac5a425faf6b" exitCode=0 Oct 31 14:11:08.379151 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:08.378663 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" event={"ID":"c0c76746-d068-475c-9ac5-856f863dac4a","Type":"ContainerDied","Data":"0cf83325a6e8fdbfa60eacf33f9863439501d5b8d9c89e5a5a27ac5a425faf6b"} Oct 31 14:11:09.524492 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:09.524467 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" Oct 31 14:11:09.613228 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:09.613193 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-openshift-ca\") pod \"c0c76746-d068-475c-9ac5-856f863dac4a\" (UID: \"c0c76746-d068-475c-9ac5-856f863dac4a\") " Oct 31 14:11:09.613424 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:09.613275 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-xks-ca\") pod \"c0c76746-d068-475c-9ac5-856f863dac4a\" (UID: \"c0c76746-d068-475c-9ac5-856f863dac4a\") " Oct 31 14:11:09.613424 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:09.613321 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c76746-d068-475c-9ac5-856f863dac4a-kube-api-access-lbpvl\" (UniqueName: \"kubernetes.io/projected/c0c76746-d068-475c-9ac5-856f863dac4a-kube-api-access-lbpvl\") pod \"c0c76746-d068-475c-9ac5-856f863dac4a\" (UID: \"c0c76746-d068-475c-9ac5-856f863dac4a\") " Oct 31 14:11:09.613681 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:09.613654 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-openshift-ca" pod "c0c76746-d068-475c-9ac5-856f863dac4a" (UID: "c0c76746-d068-475c-9ac5-856f863dac4a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:11:09.615445 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:09.615414 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-xks-ca" pod "c0c76746-d068-475c-9ac5-856f863dac4a" (UID: "c0c76746-d068-475c-9ac5-856f863dac4a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:11:09.615554 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:09.615532 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c76746-d068-475c-9ac5-856f863dac4a-kube-api-access-lbpvl" pod "c0c76746-d068-475c-9ac5-856f863dac4a" (UID: "c0c76746-d068-475c-9ac5-856f863dac4a"). InnerVolumeSpecName "kube-api-access-lbpvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:11:09.714572 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:09.714536 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:11:09.714572 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:09.714568 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c76746-d068-475c-9ac5-856f863dac4a-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:11:09.714572 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:09.714578 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbpvl\" (UniqueName: \"kubernetes.io/projected/c0c76746-d068-475c-9ac5-856f863dac4a-kube-api-access-lbpvl\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:11:10.390253 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:10.390167 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" event={"ID":"c0c76746-d068-475c-9ac5-856f863dac4a","Type":"ContainerDied","Data":"152e57f94ae75ba768ca1724d69ae12da63b248e1ced25dd62c76d300bc42d73"} Oct 31 14:11:10.390253 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:10.390225 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="152e57f94ae75ba768ca1724d69ae12da63b248e1ced25dd62c76d300bc42d73" Oct 31 14:11:10.390253 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:10.390184 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n" Oct 31 14:11:17.467150 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.467114 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb"] Oct 31 14:11:17.467813 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.467789 5532 cpu_manager.go:414] "containerMap: removing container" podUID="c0c76746-d068-475c-9ac5-856f863dac4a" containerName="revoke-keys" Oct 31 14:11:17.467938 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.467814 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c76746-d068-475c-9ac5-856f863dac4a" containerName="revoke-keys" Oct 31 14:11:17.467984 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.467967 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c76746-d068-475c-9ac5-856f863dac4a" containerName="revoke-keys" Oct 31 14:11:17.471765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.471746 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" Oct 31 14:11:17.480351 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.480321 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:11:17.480486 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.480334 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:11:17.480486 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.480436 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:11:17.480841 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.480818 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:11:17.482077 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.482045 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb"] Oct 31 14:11:17.497188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.497157 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb\" (UID: \"1bf08f38-3cf6-42fb-b346-34c915eaa197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" Oct 31 14:11:17.497352 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.497331 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb\" (UID: \"1bf08f38-3cf6-42fb-b346-34c915eaa197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" Oct 31 14:11:17.497471 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.497450 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tph7x\" (UniqueName: \"kubernetes.io/projected/1bf08f38-3cf6-42fb-b346-34c915eaa197-kube-api-access-tph7x\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb\" (UID: \"1bf08f38-3cf6-42fb-b346-34c915eaa197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" Oct 31 14:11:17.598481 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.598441 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb\" (UID: \"1bf08f38-3cf6-42fb-b346-34c915eaa197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" Oct 31 14:11:17.598738 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.598548 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb\" (UID: \"1bf08f38-3cf6-42fb-b346-34c915eaa197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" Oct 31 14:11:17.598738 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.598643 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tph7x\" (UniqueName: \"kubernetes.io/projected/1bf08f38-3cf6-42fb-b346-34c915eaa197-kube-api-access-tph7x\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb\" (UID: \"1bf08f38-3cf6-42fb-b346-34c915eaa197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" Oct 31 14:11:17.599220 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.599192 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb\" (UID: \"1bf08f38-3cf6-42fb-b346-34c915eaa197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" Oct 31 14:11:17.601086 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.601043 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb\" (UID: \"1bf08f38-3cf6-42fb-b346-34c915eaa197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" Oct 31 14:11:17.613999 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.613970 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tph7x\" (UniqueName: \"kubernetes.io/projected/1bf08f38-3cf6-42fb-b346-34c915eaa197-kube-api-access-tph7x\") pod \"maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb\" (UID: \"1bf08f38-3cf6-42fb-b346-34c915eaa197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" Oct 31 14:11:17.795159 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:17.795071 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" Oct 31 14:11:18.074965 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:18.074937 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb"] Oct 31 14:11:18.077688 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:11:18.077659 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf08f38_3cf6_42fb_b346_34c915eaa197.slice/crio-ac4d65eacd88658a0a403255c963cb772acd428151665c95cd220bc669ecdc03 WatchSource:0}: Error finding container ac4d65eacd88658a0a403255c963cb772acd428151665c95cd220bc669ecdc03: Status 404 returned error can't find the container with id ac4d65eacd88658a0a403255c963cb772acd428151665c95cd220bc669ecdc03 Oct 31 14:11:18.305754 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:18.305713 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:11:18.439929 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:18.439887 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" event={"ID":"1bf08f38-3cf6-42fb-b346-34c915eaa197","Type":"ContainerStarted","Data":"e3be061b7d1ddb645a49414fdcac526a9fbcf78168c1f912f57655770ecbf5bd"} Oct 31 14:11:18.440125 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:18.439950 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" event={"ID":"1bf08f38-3cf6-42fb-b346-34c915eaa197","Type":"ContainerStarted","Data":"ac4d65eacd88658a0a403255c963cb772acd428151665c95cd220bc669ecdc03"} Oct 31 14:11:18.464975 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:18.464923 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" podStartSLOduration=1.4649098870000001 podStartE2EDuration="1.464909887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:11:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:11:18.464117071 +0000 UTC m=+4046.609860896" watchObservedRunningTime="2026-10-31 14:11:18.464909887 +0000 UTC m=+4046.610653720" Oct 31 14:11:19.446548 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:19.446515 5532 generic.go:356] "Generic (PLEG): container finished" podID="1bf08f38-3cf6-42fb-b346-34c915eaa197" containerID="e3be061b7d1ddb645a49414fdcac526a9fbcf78168c1f912f57655770ecbf5bd" exitCode=0 Oct 31 14:11:19.447039 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:19.446629 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" event={"ID":"1bf08f38-3cf6-42fb-b346-34c915eaa197","Type":"ContainerDied","Data":"e3be061b7d1ddb645a49414fdcac526a9fbcf78168c1f912f57655770ecbf5bd"} Oct 31 14:11:20.589308 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:20.589278 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" Oct 31 14:11:20.632625 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:20.632584 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-xks-ca\") pod \"1bf08f38-3cf6-42fb-b346-34c915eaa197\" (UID: \"1bf08f38-3cf6-42fb-b346-34c915eaa197\") " Oct 31 14:11:20.632813 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:20.632781 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-openshift-ca\") pod \"1bf08f38-3cf6-42fb-b346-34c915eaa197\" (UID: \"1bf08f38-3cf6-42fb-b346-34c915eaa197\") " Oct 31 14:11:20.632813 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:20.632810 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bf08f38-3cf6-42fb-b346-34c915eaa197-kube-api-access-tph7x\" (UniqueName: \"kubernetes.io/projected/1bf08f38-3cf6-42fb-b346-34c915eaa197-kube-api-access-tph7x\") pod \"1bf08f38-3cf6-42fb-b346-34c915eaa197\" (UID: \"1bf08f38-3cf6-42fb-b346-34c915eaa197\") " Oct 31 14:11:20.633187 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:20.633160 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-openshift-ca" pod "1bf08f38-3cf6-42fb-b346-34c915eaa197" (UID: "1bf08f38-3cf6-42fb-b346-34c915eaa197"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:11:20.634819 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:20.634779 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-xks-ca" pod "1bf08f38-3cf6-42fb-b346-34c915eaa197" (UID: "1bf08f38-3cf6-42fb-b346-34c915eaa197"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:11:20.635040 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:20.635013 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf08f38-3cf6-42fb-b346-34c915eaa197-kube-api-access-tph7x" pod "1bf08f38-3cf6-42fb-b346-34c915eaa197" (UID: "1bf08f38-3cf6-42fb-b346-34c915eaa197"). InnerVolumeSpecName "kube-api-access-tph7x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:11:20.734562 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:20.734480 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:11:20.734562 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:20.734515 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tph7x\" (UniqueName: \"kubernetes.io/projected/1bf08f38-3cf6-42fb-b346-34c915eaa197-kube-api-access-tph7x\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:11:20.734562 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:20.734534 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bf08f38-3cf6-42fb-b346-34c915eaa197-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:11:21.458791 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:21.458759 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" Oct 31 14:11:21.458977 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:21.458754 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb" event={"ID":"1bf08f38-3cf6-42fb-b346-34c915eaa197","Type":"ContainerDied","Data":"ac4d65eacd88658a0a403255c963cb772acd428151665c95cd220bc669ecdc03"} Oct 31 14:11:21.458977 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:21.458905 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac4d65eacd88658a0a403255c963cb772acd428151665c95cd220bc669ecdc03" Oct 31 14:11:29.128848 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:29.128814 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb"] Oct 31 14:11:29.132842 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:29.132812 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-c4292c-gbhbb"] Oct 31 14:11:30.003302 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:30.003079 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bf08f38-3cf6-42fb-b346-34c915eaa197" path="/var/lib/kubelet/pods/1bf08f38-3cf6-42fb-b346-34c915eaa197/volumes" Oct 31 14:11:30.620689 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:30.620647 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:11:31.760461 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:11:31.760420 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:11:49.317084 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:49.317045 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:11:54.117759 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:11:54.117725 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:12:01.760743 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:12:01.760701 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:12:31.760636 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:12:31.760559 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:13:01.761243 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:13:01.761147 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:13:24.849801 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:13:24.849763 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:13:31.761531 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:13:31.761495 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:13:39.956126 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:13:39.956094 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:13:44.275944 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:13:44.275904 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:14:01.077404 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:01.077367 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:14:01.760697 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:14:01.760656 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:14:08.911393 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:08.911355 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:14:26.254770 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:26.254731 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:14:31.761649 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:14:31.761533 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:14:40.376518 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:40.376481 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:14:52.971991 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:52.971952 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m"] Oct 31 14:14:52.972425 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:52.972410 5532 cpu_manager.go:414] "containerMap: removing container" podUID="1bf08f38-3cf6-42fb-b346-34c915eaa197" containerName="revoke-keys" Oct 31 14:14:52.972522 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:52.972427 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bf08f38-3cf6-42fb-b346-34c915eaa197" containerName="revoke-keys" Oct 31 14:14:52.972561 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:52.972541 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bf08f38-3cf6-42fb-b346-34c915eaa197" containerName="revoke-keys" Oct 31 14:14:52.975920 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:52.975902 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" Oct 31 14:14:52.983881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:52.983792 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:14:52.984059 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:52.983867 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:14:52.984059 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:52.983950 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:14:52.984229 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:52.984005 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:14:52.985193 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:52.985168 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m"] Oct 31 14:14:53.054990 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:53.054951 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5qgw\" (UniqueName: \"kubernetes.io/projected/07f19ad5-3e66-4c5a-98cc-9511636a1184-kube-api-access-l5qgw\") pod \"maas-api-revoke-sub-11d9ca72-z494m\" (UID: \"07f19ad5-3e66-4c5a-98cc-9511636a1184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" Oct 31 14:14:53.055192 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:53.055107 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11d9ca72-z494m\" (UID: \"07f19ad5-3e66-4c5a-98cc-9511636a1184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" Oct 31 14:14:53.055192 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:53.055184 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11d9ca72-z494m\" (UID: \"07f19ad5-3e66-4c5a-98cc-9511636a1184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" Oct 31 14:14:53.156417 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:53.156381 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11d9ca72-z494m\" (UID: \"07f19ad5-3e66-4c5a-98cc-9511636a1184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" Oct 31 14:14:53.156643 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:53.156429 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11d9ca72-z494m\" (UID: \"07f19ad5-3e66-4c5a-98cc-9511636a1184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" Oct 31 14:14:53.156643 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:53.156521 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5qgw\" (UniqueName: \"kubernetes.io/projected/07f19ad5-3e66-4c5a-98cc-9511636a1184-kube-api-access-l5qgw\") pod \"maas-api-revoke-sub-11d9ca72-z494m\" (UID: \"07f19ad5-3e66-4c5a-98cc-9511636a1184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" Oct 31 14:14:53.157308 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:53.157282 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11d9ca72-z494m\" (UID: \"07f19ad5-3e66-4c5a-98cc-9511636a1184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" Oct 31 14:14:53.158990 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:53.158967 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11d9ca72-z494m\" (UID: \"07f19ad5-3e66-4c5a-98cc-9511636a1184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" Oct 31 14:14:53.170083 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:53.170056 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5qgw\" (UniqueName: \"kubernetes.io/projected/07f19ad5-3e66-4c5a-98cc-9511636a1184-kube-api-access-l5qgw\") pod \"maas-api-revoke-sub-11d9ca72-z494m\" (UID: \"07f19ad5-3e66-4c5a-98cc-9511636a1184\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" Oct 31 14:14:53.290933 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:53.290850 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" Oct 31 14:14:53.579962 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:53.579935 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m"] Oct 31 14:14:53.581987 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:14:53.581959 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f19ad5_3e66_4c5a_98cc_9511636a1184.slice/crio-1cb95b4ffc5d5cad3dbb6895ca7ceee86523fddf1204bf6ce7a58415f5572c84 WatchSource:0}: Error finding container 1cb95b4ffc5d5cad3dbb6895ca7ceee86523fddf1204bf6ce7a58415f5572c84: Status 404 returned error can't find the container with id 1cb95b4ffc5d5cad3dbb6895ca7ceee86523fddf1204bf6ce7a58415f5572c84 Oct 31 14:14:53.630123 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:53.630097 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" event={"ID":"07f19ad5-3e66-4c5a-98cc-9511636a1184","Type":"ContainerStarted","Data":"1cb95b4ffc5d5cad3dbb6895ca7ceee86523fddf1204bf6ce7a58415f5572c84"} Oct 31 14:14:54.637779 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:54.637745 5532 generic.go:356] "Generic (PLEG): container finished" podID="07f19ad5-3e66-4c5a-98cc-9511636a1184" containerID="98a6a90d4060990f761f604ef93ecf42dae533949f47ca4055aef19e9e123302" exitCode=0 Oct 31 14:14:54.638221 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:54.637831 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" event={"ID":"07f19ad5-3e66-4c5a-98cc-9511636a1184","Type":"ContainerDied","Data":"98a6a90d4060990f761f604ef93ecf42dae533949f47ca4055aef19e9e123302"} Oct 31 14:14:55.780960 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:55.780934 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" Oct 31 14:14:55.884518 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:55.884483 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f19ad5-3e66-4c5a-98cc-9511636a1184-kube-api-access-l5qgw\" (UniqueName: \"kubernetes.io/projected/07f19ad5-3e66-4c5a-98cc-9511636a1184-kube-api-access-l5qgw\") pod \"07f19ad5-3e66-4c5a-98cc-9511636a1184\" (UID: \"07f19ad5-3e66-4c5a-98cc-9511636a1184\") " Oct 31 14:14:55.884735 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:55.884543 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-xks-ca\") pod \"07f19ad5-3e66-4c5a-98cc-9511636a1184\" (UID: \"07f19ad5-3e66-4c5a-98cc-9511636a1184\") " Oct 31 14:14:55.884735 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:55.884693 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-openshift-ca\") pod \"07f19ad5-3e66-4c5a-98cc-9511636a1184\" (UID: \"07f19ad5-3e66-4c5a-98cc-9511636a1184\") " Oct 31 14:14:55.885091 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:55.885062 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-openshift-ca" pod "07f19ad5-3e66-4c5a-98cc-9511636a1184" (UID: "07f19ad5-3e66-4c5a-98cc-9511636a1184"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:14:55.886750 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:55.886721 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-xks-ca" pod "07f19ad5-3e66-4c5a-98cc-9511636a1184" (UID: "07f19ad5-3e66-4c5a-98cc-9511636a1184"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:14:55.886842 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:55.886767 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f19ad5-3e66-4c5a-98cc-9511636a1184-kube-api-access-l5qgw" pod "07f19ad5-3e66-4c5a-98cc-9511636a1184" (UID: "07f19ad5-3e66-4c5a-98cc-9511636a1184"). InnerVolumeSpecName "kube-api-access-l5qgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:14:55.986105 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:55.986022 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:14:55.986105 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:55.986053 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5qgw\" (UniqueName: \"kubernetes.io/projected/07f19ad5-3e66-4c5a-98cc-9511636a1184-kube-api-access-l5qgw\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:14:55.986105 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:55.986069 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07f19ad5-3e66-4c5a-98cc-9511636a1184-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:14:56.651440 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:56.651412 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" Oct 31 14:14:56.651654 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:56.651441 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m" event={"ID":"07f19ad5-3e66-4c5a-98cc-9511636a1184","Type":"ContainerDied","Data":"1cb95b4ffc5d5cad3dbb6895ca7ceee86523fddf1204bf6ce7a58415f5572c84"} Oct 31 14:14:56.651654 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:14:56.651491 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cb95b4ffc5d5cad3dbb6895ca7ceee86523fddf1204bf6ce7a58415f5572c84" Oct 31 14:15:00.135345 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.135305 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck"] Oct 31 14:15:00.135833 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.135821 5532 cpu_manager.go:414] "containerMap: removing container" podUID="07f19ad5-3e66-4c5a-98cc-9511636a1184" containerName="revoke-keys" Oct 31 14:15:00.135888 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.135835 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f19ad5-3e66-4c5a-98cc-9511636a1184" containerName="revoke-keys" Oct 31 14:15:00.135925 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.135900 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f19ad5-3e66-4c5a-98cc-9511636a1184" containerName="revoke-keys" Oct 31 14:15:00.139046 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.139025 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" Oct 31 14:15:00.148325 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.148299 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck"] Oct 31 14:15:00.151386 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.151358 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hxdwv\"" Oct 31 14:15:00.151533 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.151455 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:15:00.151533 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.151510 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:15:00.327083 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.327040 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h54f\" (UniqueName: \"kubernetes.io/projected/881eecce-46cd-4bae-8f02-667144b97293-kube-api-access-9h54f\") pod \"maas-api-key-cleanup-29890935-nvwck\" (UID: \"881eecce-46cd-4bae-8f02-667144b97293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" Oct 31 14:15:00.327298 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.327170 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/881eecce-46cd-4bae-8f02-667144b97293-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890935-nvwck\" (UID: \"881eecce-46cd-4bae-8f02-667144b97293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" Oct 31 14:15:00.428805 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.428764 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/881eecce-46cd-4bae-8f02-667144b97293-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890935-nvwck\" (UID: \"881eecce-46cd-4bae-8f02-667144b97293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" Oct 31 14:15:00.429022 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.428920 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h54f\" (UniqueName: \"kubernetes.io/projected/881eecce-46cd-4bae-8f02-667144b97293-kube-api-access-9h54f\") pod \"maas-api-key-cleanup-29890935-nvwck\" (UID: \"881eecce-46cd-4bae-8f02-667144b97293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" Oct 31 14:15:00.429507 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.429485 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/881eecce-46cd-4bae-8f02-667144b97293-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890935-nvwck\" (UID: \"881eecce-46cd-4bae-8f02-667144b97293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" Oct 31 14:15:00.445308 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.445279 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h54f\" (UniqueName: \"kubernetes.io/projected/881eecce-46cd-4bae-8f02-667144b97293-kube-api-access-9h54f\") pod \"maas-api-key-cleanup-29890935-nvwck\" (UID: \"881eecce-46cd-4bae-8f02-667144b97293\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" Oct 31 14:15:00.456118 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.456092 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" Oct 31 14:15:00.739469 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:00.739398 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck"] Oct 31 14:15:00.743555 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:15:00.743519 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881eecce_46cd_4bae_8f02_667144b97293.slice/crio-615d43d1e2f4067c5b65a8f2beae5772700c6c51bfa25fd2e7cd32b1b7405632 WatchSource:0}: Error finding container 615d43d1e2f4067c5b65a8f2beae5772700c6c51bfa25fd2e7cd32b1b7405632: Status 404 returned error can't find the container with id 615d43d1e2f4067c5b65a8f2beae5772700c6c51bfa25fd2e7cd32b1b7405632 Oct 31 14:15:01.680511 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:01.680452 5532 generic.go:356] "Generic (PLEG): container finished" podID="881eecce-46cd-4bae-8f02-667144b97293" containerID="217351228c7d02c87c91c177075b6d6ba2c3ed10179ba44dd4586ef81648db82" exitCode=60 Oct 31 14:15:01.680937 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:01.680534 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" event={"ID":"881eecce-46cd-4bae-8f02-667144b97293","Type":"ContainerDied","Data":"217351228c7d02c87c91c177075b6d6ba2c3ed10179ba44dd4586ef81648db82"} Oct 31 14:15:01.680937 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:01.680657 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" event={"ID":"881eecce-46cd-4bae-8f02-667144b97293","Type":"ContainerStarted","Data":"615d43d1e2f4067c5b65a8f2beae5772700c6c51bfa25fd2e7cd32b1b7405632"} Oct 31 14:15:01.680937 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:01.680935 5532 scope.go:122] "RemoveContainer" containerID="217351228c7d02c87c91c177075b6d6ba2c3ed10179ba44dd4586ef81648db82" Oct 31 14:15:01.762851 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:15:01.762816 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:15:02.687444 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:02.687401 5532 generic.go:356] "Generic (PLEG): container finished" podID="881eecce-46cd-4bae-8f02-667144b97293" containerID="e67162929bc0558f649f1b5e61d5342bf45a46e68152d5f2941e286a799a37cb" exitCode=60 Oct 31 14:15:02.687444 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:02.687440 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" event={"ID":"881eecce-46cd-4bae-8f02-667144b97293","Type":"ContainerDied","Data":"e67162929bc0558f649f1b5e61d5342bf45a46e68152d5f2941e286a799a37cb"} Oct 31 14:15:02.687978 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:02.687508 5532 scope.go:122] "RemoveContainer" containerID="217351228c7d02c87c91c177075b6d6ba2c3ed10179ba44dd4586ef81648db82" Oct 31 14:15:02.687978 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:02.687784 5532 scope.go:122] "RemoveContainer" containerID="e67162929bc0558f649f1b5e61d5342bf45a46e68152d5f2941e286a799a37cb" Oct 31 14:15:02.688093 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:15:02.688067 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890935-nvwck_odh-ai-gateway-infra(881eecce-46cd-4bae-8f02-667144b97293)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" podUID="881eecce-46cd-4bae-8f02-667144b97293" Oct 31 14:15:03.695977 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:03.695936 5532 scope.go:122] "RemoveContainer" containerID="e67162929bc0558f649f1b5e61d5342bf45a46e68152d5f2941e286a799a37cb" Oct 31 14:15:03.696400 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:15:03.696213 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890935-nvwck_odh-ai-gateway-infra(881eecce-46cd-4bae-8f02-667144b97293)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" podUID="881eecce-46cd-4bae-8f02-667144b97293" Oct 31 14:15:03.926172 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:03.926139 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr"] Oct 31 14:15:03.931010 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:03.930989 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" Oct 31 14:15:03.938764 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:03.938741 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 14:15:03.940172 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:03.940151 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr"] Oct 31 14:15:03.966069 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:03.965990 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr\" (UID: \"938099c5-b2a8-487c-962f-b63dace5452c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" Oct 31 14:15:03.966069 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:03.966040 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr\" (UID: \"938099c5-b2a8-487c-962f-b63dace5452c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" Oct 31 14:15:03.966254 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:03.966194 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc756\" (UniqueName: \"kubernetes.io/projected/938099c5-b2a8-487c-962f-b63dace5452c-kube-api-access-rc756\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr\" (UID: \"938099c5-b2a8-487c-962f-b63dace5452c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" Oct 31 14:15:04.067152 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:04.067111 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc756\" (UniqueName: \"kubernetes.io/projected/938099c5-b2a8-487c-962f-b63dace5452c-kube-api-access-rc756\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr\" (UID: \"938099c5-b2a8-487c-962f-b63dace5452c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" Oct 31 14:15:04.067364 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:04.067172 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr\" (UID: \"938099c5-b2a8-487c-962f-b63dace5452c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" Oct 31 14:15:04.067364 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:04.067208 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr\" (UID: \"938099c5-b2a8-487c-962f-b63dace5452c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" Oct 31 14:15:04.067902 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:04.067879 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr\" (UID: \"938099c5-b2a8-487c-962f-b63dace5452c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" Oct 31 14:15:04.069812 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:04.069788 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr\" (UID: \"938099c5-b2a8-487c-962f-b63dace5452c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" Oct 31 14:15:04.113473 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:04.113440 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc756\" (UniqueName: \"kubernetes.io/projected/938099c5-b2a8-487c-962f-b63dace5452c-kube-api-access-rc756\") pod \"maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr\" (UID: \"938099c5-b2a8-487c-962f-b63dace5452c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" Oct 31 14:15:04.245648 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:04.245549 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" Oct 31 14:15:04.322730 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:04.322701 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:15:04.530021 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:04.529996 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr"] Oct 31 14:15:04.532385 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:15:04.532359 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod938099c5_b2a8_487c_962f_b63dace5452c.slice/crio-f0d676b1311a18dd89e684e0ca47434b05beaf13636532113bc0a1c015bb14f5 WatchSource:0}: Error finding container f0d676b1311a18dd89e684e0ca47434b05beaf13636532113bc0a1c015bb14f5: Status 404 returned error can't find the container with id f0d676b1311a18dd89e684e0ca47434b05beaf13636532113bc0a1c015bb14f5 Oct 31 14:15:04.702113 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:04.702074 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" event={"ID":"938099c5-b2a8-487c-962f-b63dace5452c","Type":"ContainerStarted","Data":"f0d676b1311a18dd89e684e0ca47434b05beaf13636532113bc0a1c015bb14f5"} Oct 31 14:15:05.708178 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:05.708144 5532 generic.go:356] "Generic (PLEG): container finished" podID="938099c5-b2a8-487c-962f-b63dace5452c" containerID="c698b83f12b753d2da363bd6bb92178fce779765887f6a5279a1492700ae76b4" exitCode=0 Oct 31 14:15:05.708588 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:05.708211 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" event={"ID":"938099c5-b2a8-487c-962f-b63dace5452c","Type":"ContainerDied","Data":"c698b83f12b753d2da363bd6bb92178fce779765887f6a5279a1492700ae76b4"} Oct 31 14:15:06.850563 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:06.850535 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" Oct 31 14:15:06.897693 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:06.897659 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-xks-ca\") pod \"938099c5-b2a8-487c-962f-b63dace5452c\" (UID: \"938099c5-b2a8-487c-962f-b63dace5452c\") " Oct 31 14:15:06.897881 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:06.897839 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/938099c5-b2a8-487c-962f-b63dace5452c-kube-api-access-rc756\" (UniqueName: \"kubernetes.io/projected/938099c5-b2a8-487c-962f-b63dace5452c-kube-api-access-rc756\") pod \"938099c5-b2a8-487c-962f-b63dace5452c\" (UID: \"938099c5-b2a8-487c-962f-b63dace5452c\") " Oct 31 14:15:06.897950 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:06.897918 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-openshift-ca\") pod \"938099c5-b2a8-487c-962f-b63dace5452c\" (UID: \"938099c5-b2a8-487c-962f-b63dace5452c\") " Oct 31 14:15:06.898335 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:06.898308 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-openshift-ca" pod "938099c5-b2a8-487c-962f-b63dace5452c" (UID: "938099c5-b2a8-487c-962f-b63dace5452c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:15:06.899795 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:06.899773 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-xks-ca" pod "938099c5-b2a8-487c-962f-b63dace5452c" (UID: "938099c5-b2a8-487c-962f-b63dace5452c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 14:15:06.900096 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:06.900074 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/938099c5-b2a8-487c-962f-b63dace5452c-kube-api-access-rc756" pod "938099c5-b2a8-487c-962f-b63dace5452c" (UID: "938099c5-b2a8-487c-962f-b63dace5452c"). InnerVolumeSpecName "kube-api-access-rc756". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:15:06.999045 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:06.998963 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc756\" (UniqueName: \"kubernetes.io/projected/938099c5-b2a8-487c-962f-b63dace5452c-kube-api-access-rc756\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:15:06.999045 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:06.998991 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-openshift-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:15:06.999045 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:06.999001 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/938099c5-b2a8-487c-962f-b63dace5452c-maas-api-xks-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:15:07.719739 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:07.719693 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" event={"ID":"938099c5-b2a8-487c-962f-b63dace5452c","Type":"ContainerDied","Data":"f0d676b1311a18dd89e684e0ca47434b05beaf13636532113bc0a1c015bb14f5"} Oct 31 14:15:07.719927 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:07.719779 5532 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0d676b1311a18dd89e684e0ca47434b05beaf13636532113bc0a1c015bb14f5" Oct 31 14:15:07.719927 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:07.719706 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr" Oct 31 14:15:12.996031 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:12.995991 5532 scope.go:122] "RemoveContainer" containerID="e67162929bc0558f649f1b5e61d5342bf45a46e68152d5f2941e286a799a37cb" Oct 31 14:15:13.753188 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:13.753147 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" event={"ID":"881eecce-46cd-4bae-8f02-667144b97293","Type":"ContainerStarted","Data":"bda4ad91a07a55b7c0a01e04093ae80004abd2aa363d3af65805357d3d144812"} Oct 31 14:15:13.775473 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:13.775423 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" podStartSLOduration=13.775410472 podStartE2EDuration="13.775410472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:15:13.773890205 +0000 UTC m=+4281.919634034" watchObservedRunningTime="2026-10-31 14:15:13.775410472 +0000 UTC m=+4281.921154300" Oct 31 14:15:14.759765 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:14.759733 5532 generic.go:356] "Generic (PLEG): container finished" podID="881eecce-46cd-4bae-8f02-667144b97293" containerID="bda4ad91a07a55b7c0a01e04093ae80004abd2aa363d3af65805357d3d144812" exitCode=60 Oct 31 14:15:14.760202 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:14.759812 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" event={"ID":"881eecce-46cd-4bae-8f02-667144b97293","Type":"ContainerDied","Data":"bda4ad91a07a55b7c0a01e04093ae80004abd2aa363d3af65805357d3d144812"} Oct 31 14:15:14.760202 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:14.759886 5532 scope.go:122] "RemoveContainer" containerID="e67162929bc0558f649f1b5e61d5342bf45a46e68152d5f2941e286a799a37cb" Oct 31 14:15:14.760303 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:14.760237 5532 scope.go:122] "RemoveContainer" containerID="bda4ad91a07a55b7c0a01e04093ae80004abd2aa363d3af65805357d3d144812" Oct 31 14:15:14.760520 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:15:14.760500 5532 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890935-nvwck_odh-ai-gateway-infra(881eecce-46cd-4bae-8f02-667144b97293)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" podUID="881eecce-46cd-4bae-8f02-667144b97293" Oct 31 14:15:14.796810 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:14.796783 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck"] Oct 31 14:15:15.169175 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:15.169146 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr"] Oct 31 14:15:15.173195 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:15.173162 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-a9dd7f-kzfhr"] Oct 31 14:15:15.903473 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:15.903448 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" Oct 31 14:15:15.996390 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:15.996350 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881eecce-46cd-4bae-8f02-667144b97293-kube-api-access-9h54f\" (UniqueName: \"kubernetes.io/projected/881eecce-46cd-4bae-8f02-667144b97293-kube-api-access-9h54f\") pod \"881eecce-46cd-4bae-8f02-667144b97293\" (UID: \"881eecce-46cd-4bae-8f02-667144b97293\") " Oct 31 14:15:15.996613 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:15.996473 5532 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/881eecce-46cd-4bae-8f02-667144b97293-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/881eecce-46cd-4bae-8f02-667144b97293-maas-api-service-ca\") pod \"881eecce-46cd-4bae-8f02-667144b97293\" (UID: \"881eecce-46cd-4bae-8f02-667144b97293\") " Oct 31 14:15:15.997003 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:15.996973 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881eecce-46cd-4bae-8f02-667144b97293-maas-api-service-ca" pod "881eecce-46cd-4bae-8f02-667144b97293" (UID: "881eecce-46cd-4bae-8f02-667144b97293"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 14:15:15.997148 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:15.997132 5532 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/881eecce-46cd-4bae-8f02-667144b97293-maas-api-service-ca\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:15:15.998572 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:15.998548 5532 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881eecce-46cd-4bae-8f02-667144b97293-kube-api-access-9h54f" pod "881eecce-46cd-4bae-8f02-667144b97293" (UID: "881eecce-46cd-4bae-8f02-667144b97293"). InnerVolumeSpecName "kube-api-access-9h54f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 14:15:16.003673 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:16.003643 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="938099c5-b2a8-487c-962f-b63dace5452c" path="/var/lib/kubelet/pods/938099c5-b2a8-487c-962f-b63dace5452c/volumes" Oct 31 14:15:16.098441 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:16.098367 5532 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9h54f\" (UniqueName: \"kubernetes.io/projected/881eecce-46cd-4bae-8f02-667144b97293-kube-api-access-9h54f\") on node \"ip-10-0-128-41.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 14:15:16.777995 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:16.777967 5532 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" Oct 31 14:15:16.778172 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:16.777995 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck" event={"ID":"881eecce-46cd-4bae-8f02-667144b97293","Type":"ContainerDied","Data":"615d43d1e2f4067c5b65a8f2beae5772700c6c51bfa25fd2e7cd32b1b7405632"} Oct 31 14:15:16.778172 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:16.778077 5532 scope.go:122] "RemoveContainer" containerID="bda4ad91a07a55b7c0a01e04093ae80004abd2aa363d3af65805357d3d144812" Oct 31 14:15:16.800949 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:16.800926 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck"] Oct 31 14:15:16.806306 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:16.806281 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890935-nvwck"] Oct 31 14:15:16.835899 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:16.835871 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:15:18.001472 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:18.001437 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881eecce-46cd-4bae-8f02-667144b97293" path="/var/lib/kubelet/pods/881eecce-46cd-4bae-8f02-667144b97293/volumes" Oct 31 14:15:29.656359 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:29.656321 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:15:31.888092 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:15:31.888046 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:15:46.212965 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:15:46.212929 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:16:01.224154 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:01.224070 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:16:01.764904 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:16:01.764865 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:16:10.027819 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:10.027783 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n"] Oct 31 14:16:10.032228 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:10.032200 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd0c3124-42v4n"] Oct 31 14:16:12.001854 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:12.001822 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c76746-d068-475c-9ac5-856f863dac4a" path="/var/lib/kubelet/pods/c0c76746-d068-475c-9ac5-856f863dac4a/volumes" Oct 31 14:16:13.025419 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:13.025386 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:16:19.063935 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:19.063880 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:16:27.660101 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:27.660067 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:16:31.906619 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:16:31.905082 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:16:31.906619 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:31.906011 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:16:32.128761 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:32.128720 5532 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 14:16:32.129119 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:32.129085 5532 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5d84bc6b95-hdtr7" podUID="b13b4423-6b52-4dd7-be75-987fd6e2edfa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 14:16:49.254782 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:49.254749 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:16:58.711406 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.711371 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-64fbfb966c-v944b"] Oct 31 14:16:58.711949 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.711817 5532 cpu_manager.go:414] "containerMap: removing container" podUID="881eecce-46cd-4bae-8f02-667144b97293" containerName="cleanup" Oct 31 14:16:58.711949 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.711831 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881eecce-46cd-4bae-8f02-667144b97293" containerName="cleanup" Oct 31 14:16:58.711949 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.711840 5532 cpu_manager.go:414] "containerMap: removing container" podUID="938099c5-b2a8-487c-962f-b63dace5452c" containerName="revoke-keys" Oct 31 14:16:58.711949 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.711845 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="938099c5-b2a8-487c-962f-b63dace5452c" containerName="revoke-keys" Oct 31 14:16:58.711949 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.711869 5532 cpu_manager.go:414] "containerMap: removing container" podUID="881eecce-46cd-4bae-8f02-667144b97293" containerName="cleanup" Oct 31 14:16:58.711949 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.711874 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881eecce-46cd-4bae-8f02-667144b97293" containerName="cleanup" Oct 31 14:16:58.711949 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.711891 5532 cpu_manager.go:414] "containerMap: removing container" podUID="881eecce-46cd-4bae-8f02-667144b97293" containerName="cleanup" Oct 31 14:16:58.711949 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.711896 5532 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881eecce-46cd-4bae-8f02-667144b97293" containerName="cleanup" Oct 31 14:16:58.712224 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.711958 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="881eecce-46cd-4bae-8f02-667144b97293" containerName="cleanup" Oct 31 14:16:58.712224 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.711966 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="938099c5-b2a8-487c-962f-b63dace5452c" containerName="revoke-keys" Oct 31 14:16:58.712224 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.711977 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="881eecce-46cd-4bae-8f02-667144b97293" containerName="cleanup" Oct 31 14:16:58.715033 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.715011 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:58.722226 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.722198 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 14:16:58.729283 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.729250 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64fbfb966c-v944b"] Oct 31 14:16:58.741238 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.741214 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2284b8d-a107-41cc-a881-b9b9ee363da7-metrics-certs\") pod \"maas-api-64fbfb966c-v944b\" (UID: \"a2284b8d-a107-41cc-a881-b9b9ee363da7\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:58.741371 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.741251 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2284b8d-a107-41cc-a881-b9b9ee363da7-maas-api-tls\") pod \"maas-api-64fbfb966c-v944b\" (UID: \"a2284b8d-a107-41cc-a881-b9b9ee363da7\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:58.741371 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.741275 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cppg\" (UniqueName: \"kubernetes.io/projected/a2284b8d-a107-41cc-a881-b9b9ee363da7-kube-api-access-6cppg\") pod \"maas-api-64fbfb966c-v944b\" (UID: \"a2284b8d-a107-41cc-a881-b9b9ee363da7\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:58.741473 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.741446 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2284b8d-a107-41cc-a881-b9b9ee363da7-openshift-service-ca\") pod \"maas-api-64fbfb966c-v944b\" (UID: \"a2284b8d-a107-41cc-a881-b9b9ee363da7\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:58.742640 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.742620 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 14:16:58.742877 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.742855 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-bx4t6\"" Oct 31 14:16:58.743753 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.743735 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 14:16:58.819620 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.819567 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 14:16:58.843135 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.843102 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2284b8d-a107-41cc-a881-b9b9ee363da7-metrics-certs\") pod \"maas-api-64fbfb966c-v944b\" (UID: \"a2284b8d-a107-41cc-a881-b9b9ee363da7\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:58.843298 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.843142 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2284b8d-a107-41cc-a881-b9b9ee363da7-maas-api-tls\") pod \"maas-api-64fbfb966c-v944b\" (UID: \"a2284b8d-a107-41cc-a881-b9b9ee363da7\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:58.843298 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.843165 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cppg\" (UniqueName: \"kubernetes.io/projected/a2284b8d-a107-41cc-a881-b9b9ee363da7-kube-api-access-6cppg\") pod \"maas-api-64fbfb966c-v944b\" (UID: \"a2284b8d-a107-41cc-a881-b9b9ee363da7\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:58.843298 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.843199 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2284b8d-a107-41cc-a881-b9b9ee363da7-openshift-service-ca\") pod \"maas-api-64fbfb966c-v944b\" (UID: \"a2284b8d-a107-41cc-a881-b9b9ee363da7\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:58.844015 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.843991 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2284b8d-a107-41cc-a881-b9b9ee363da7-openshift-service-ca\") pod \"maas-api-64fbfb966c-v944b\" (UID: \"a2284b8d-a107-41cc-a881-b9b9ee363da7\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:58.845768 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.845748 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2284b8d-a107-41cc-a881-b9b9ee363da7-maas-api-tls\") pod \"maas-api-64fbfb966c-v944b\" (UID: \"a2284b8d-a107-41cc-a881-b9b9ee363da7\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:58.845856 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.845791 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2284b8d-a107-41cc-a881-b9b9ee363da7-metrics-certs\") pod \"maas-api-64fbfb966c-v944b\" (UID: \"a2284b8d-a107-41cc-a881-b9b9ee363da7\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:58.856834 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:58.856815 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cppg\" (UniqueName: \"kubernetes.io/projected/a2284b8d-a107-41cc-a881-b9b9ee363da7-kube-api-access-6cppg\") pod \"maas-api-64fbfb966c-v944b\" (UID: \"a2284b8d-a107-41cc-a881-b9b9ee363da7\") " pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:59.029899 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:59.029818 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:16:59.305861 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:59.305830 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64fbfb966c-v944b"] Oct 31 14:16:59.308208 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:16:59.308181 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2284b8d_a107_41cc_a881_b9b9ee363da7.slice/crio-a07b6c651f5237180f70b7910398cb022164f4a5e46fcb7cc34a810ca7775e09 WatchSource:0}: Error finding container a07b6c651f5237180f70b7910398cb022164f4a5e46fcb7cc34a810ca7775e09: Status 404 returned error can't find the container with id a07b6c651f5237180f70b7910398cb022164f4a5e46fcb7cc34a810ca7775e09 Oct 31 14:16:59.309623 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:59.309605 5532 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 14:16:59.546279 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:16:59.546240 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" event={"ID":"a2284b8d-a107-41cc-a881-b9b9ee363da7","Type":"ContainerStarted","Data":"a07b6c651f5237180f70b7910398cb022164f4a5e46fcb7cc34a810ca7775e09"} Oct 31 14:17:00.553513 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:17:00.553475 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" event={"ID":"a2284b8d-a107-41cc-a881-b9b9ee363da7","Type":"ContainerStarted","Data":"bc1411f585af97ac15948231abaeae5c78961208d0de466c617a2d984ec54e4e"} Oct 31 14:17:00.553994 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:17:00.553589 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:17:00.577322 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:17:00.577269 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" podStartSLOduration=2.170057583 podStartE2EDuration="2.577251252s" totalImagesPullingTime="407.193669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 14:16:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:17:00.575558828 +0000 UTC m=+4388.721302667" watchObservedRunningTime="2026-10-31 14:17:00.577251252 +0000 UTC m=+4388.722995080" Oct 31 14:17:01.761234 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:17:01.761196 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:17:05.440861 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:17:05.440823 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:17:06.563669 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:17:06.563636 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-64fbfb966c-v944b" Oct 31 14:17:29.525553 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:17:29.525515 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:17:31.760441 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:17:31.760399 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:17:43.334483 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:17:43.334447 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:17:57.693460 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:17:57.693428 5532 scope.go:122] "RemoveContainer" containerID="e3be061b7d1ddb645a49414fdcac526a9fbcf78168c1f912f57655770ecbf5bd" Oct 31 14:17:57.702590 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:17:57.702562 5532 scope.go:122] "RemoveContainer" containerID="0cf83325a6e8fdbfa60eacf33f9863439501d5b8d9c89e5a5a27ac5a425faf6b" Oct 31 14:18:01.760868 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:18:01.760828 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:18:31.761780 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:18:31.761740 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:19:01.760741 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:19:01.760658 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:19:31.760853 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:19:31.760811 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:19:32.238028 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:19:32.237994 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:19:40.233945 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:19:40.233912 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:19:45.464611 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:19:45.464573 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:19:56.038807 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:19:56.038774 5532 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m"] Oct 31 14:19:56.042300 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:19:56.042154 5532 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11d9ca72-z494m"] Oct 31 14:19:58.002152 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:19:58.002116 5532 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f19ad5-3e66-4c5a-98cc-9511636a1184" path="/var/lib/kubelet/pods/07f19ad5-3e66-4c5a-98cc-9511636a1184/volumes" Oct 31 14:20:00.553510 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:00.553475 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:20:01.761551 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:20:01.761518 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:20:27.454617 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:27.454569 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-sbgmp"] Oct 31 14:20:31.759833 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:20:31.759742 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:20:39.519950 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:39.519911 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56949457b6-sffrf_1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0/manager/0.log" Oct 31 14:20:41.136792 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:41.136753 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-64fbfb966c-v944b_a2284b8d-a107-41cc-a881-b9b9ee363da7/maas-api/0.log" Oct 31 14:20:44.929713 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:44.929681 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-sbgmp_4187620e-9bb5-401e-a28e-a0e464c82530/limitador/0.log" Oct 31 14:20:46.466655 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:46.466617 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7485898565-kp66f_c87eda91-1ec7-42bf-bf88-250b804bd332/kube-auth-proxy/0.log" Oct 31 14:20:47.225115 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:47.225071 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-76bb5_7d461f31-7477-4c4f-81fa-6a4769e2266f/istio-proxy/0.log" Oct 31 14:20:47.753575 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:47.753533 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-hdtr7_b13b4423-6b52-4dd7-be75-987fd6e2edfa/istio-proxy/0.log" Oct 31 14:20:48.130724 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:48.130643 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-qhgv4_fdf3bdf7-1bcc-43e5-bccd-a375cce224fd/istio-proxy/0.log" Oct 31 14:20:48.316834 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:48.316805 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-tvm4p_532959bd-96e4-40f7-a157-7302d6216a41/istio-proxy/0.log" Oct 31 14:20:50.396835 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:50.396805 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-h4g6w_3f952465-49ad-43cf-9598-dd3741e8de2c/main/0.log" Oct 31 14:20:51.107994 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:51.107966 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-sngft_0b01eeee-1f42-4e8e-af23-4e228345c60f/main/0.log" Oct 31 14:20:51.293218 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:51.293170 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264_e5001cfd-6512-441f-a9d9-0b566d4f06b2/main/0.log" Oct 31 14:20:57.753194 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:20:57.753160 5532 scope.go:122] "RemoveContainer" containerID="98a6a90d4060990f761f604ef93ecf42dae533949f47ca4055aef19e9e123302" Oct 31 14:21:01.760772 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:21:01.760732 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:21:02.229707 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:02.229664 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56949457b6-sffrf_1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0/manager/0.log" Oct 31 14:21:03.518860 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:03.518821 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-64fbfb966c-v944b_a2284b8d-a107-41cc-a881-b9b9ee363da7/maas-api/0.log" Oct 31 14:21:06.885167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:06.885132 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-sbgmp_4187620e-9bb5-401e-a28e-a0e464c82530/limitador/0.log" Oct 31 14:21:08.133564 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:08.133533 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7485898565-kp66f_c87eda91-1ec7-42bf-bf88-250b804bd332/kube-auth-proxy/0.log" Oct 31 14:21:08.963258 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:08.963226 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-76bb5_7d461f31-7477-4c4f-81fa-6a4769e2266f/istio-proxy/0.log" Oct 31 14:21:09.538101 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:09.538071 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-hdtr7_b13b4423-6b52-4dd7-be75-987fd6e2edfa/istio-proxy/0.log" Oct 31 14:21:09.938223 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:09.938187 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-qhgv4_fdf3bdf7-1bcc-43e5-bccd-a375cce224fd/istio-proxy/0.log" Oct 31 14:21:10.128880 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:10.128850 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-tvm4p_532959bd-96e4-40f7-a157-7302d6216a41/istio-proxy/0.log" Oct 31 14:21:11.763070 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:11.763042 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-h4g6w_3f952465-49ad-43cf-9598-dd3741e8de2c/main/0.log" Oct 31 14:21:12.492541 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:12.492512 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-sngft_0b01eeee-1f42-4e8e-af23-4e228345c60f/main/0.log" Oct 31 14:21:12.676817 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:12.676781 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-sz264_e5001cfd-6512-441f-a9d9-0b566d4f06b2/main/0.log" Oct 31 14:21:31.761023 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:21:31.760980 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:21:49.995092 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:49.995055 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56949457b6-sffrf_1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0/manager/0.log" Oct 31 14:21:50.263167 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:50.263085 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56949457b6-sffrf_1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0/manager/0.log" Oct 31 14:21:57.773376 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:21:57.773344 5532 scope.go:122] "RemoveContainer" containerID="c698b83f12b753d2da363bd6bb92178fce779765887f6a5279a1492700ae76b4" Oct 31 14:22:01.760920 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:22:01.760837 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:22:31.760687 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:22:31.760648 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:22:36.542196 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:36.542163 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56949457b6-sffrf_1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0/manager/0.log" Oct 31 14:22:36.815252 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:36.815167 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-56949457b6-sffrf_1a8ac6e8-9c7c-4370-a5a3-0ca659c44da0/manager/0.log" Oct 31 14:22:41.347153 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:41.347119 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-48bm7_2aee367b-5b87-4bfe-9620-5adf75e25b63/global-pull-secret-syncer/0.log" Oct 31 14:22:41.819514 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:41.819478 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rmt2g_7d41cf67-3992-4b1f-8d1d-4e7bd87a01e1/konnectivity-agent/0.log" Oct 31 14:22:41.852282 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:41.852251 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-41.us-east-2.compute.internal_16434a8e882a2ee35299274e3b318497/haproxy/0.log" Oct 31 14:22:50.102251 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:50.102217 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-sbgmp_4187620e-9bb5-401e-a28e-a0e464c82530/limitador/0.log" Oct 31 14:22:51.203166 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.203136 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1ae85c-6bcf-43cc-9709-3102e4c5bc59/alertmanager/0.log" Oct 31 14:22:51.229552 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.229526 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1ae85c-6bcf-43cc-9709-3102e4c5bc59/config-reloader/0.log" Oct 31 14:22:51.256492 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.256470 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1ae85c-6bcf-43cc-9709-3102e4c5bc59/kube-rbac-proxy-web/0.log" Oct 31 14:22:51.283071 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.283044 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1ae85c-6bcf-43cc-9709-3102e4c5bc59/kube-rbac-proxy/0.log" Oct 31 14:22:51.310708 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.310681 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1ae85c-6bcf-43cc-9709-3102e4c5bc59/kube-rbac-proxy-metric/0.log" Oct 31 14:22:51.338100 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.338078 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1ae85c-6bcf-43cc-9709-3102e4c5bc59/prom-label-proxy/0.log" Oct 31 14:22:51.364312 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.364291 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_bd1ae85c-6bcf-43cc-9709-3102e4c5bc59/init-config-reloader/0.log" Oct 31 14:22:51.420907 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.420876 5532 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c"] Oct 31 14:22:51.421411 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.421398 5532 memory_manager.go:365] "RemoveStaleState removing state" podUID="881eecce-46cd-4bae-8f02-667144b97293" containerName="cleanup" Oct 31 14:22:51.424447 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.424432 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.429113 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.429092 5532 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bkxgk\"/\"default-dockercfg-smppd\"" Oct 31 14:22:51.430273 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.430252 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bkxgk\"/\"kube-root-ca.crt\"" Oct 31 14:22:51.430372 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.430337 5532 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bkxgk\"/\"openshift-service-ca.crt\"" Oct 31 14:22:51.437880 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.437858 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c"] Oct 31 14:22:51.595141 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.595045 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6254fb31-ae08-438e-93cc-848b5650dfec-podres\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.595141 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.595107 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwq79\" (UniqueName: \"kubernetes.io/projected/6254fb31-ae08-438e-93cc-848b5650dfec-kube-api-access-fwq79\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.595403 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.595187 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6254fb31-ae08-438e-93cc-848b5650dfec-sys\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.595403 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.595240 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6254fb31-ae08-438e-93cc-848b5650dfec-proc\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.595403 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.595294 5532 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6254fb31-ae08-438e-93cc-848b5650dfec-lib-modules\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.596061 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.596010 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-wzvz2_849c2ed1-3d50-4936-a2a9-e734e01108e1/monitoring-plugin/0.log" Oct 31 14:22:51.696123 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.696088 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6254fb31-ae08-438e-93cc-848b5650dfec-proc\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.696302 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.696128 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6254fb31-ae08-438e-93cc-848b5650dfec-lib-modules\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.696302 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.696217 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/6254fb31-ae08-438e-93cc-848b5650dfec-proc\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.696302 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.696255 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6254fb31-ae08-438e-93cc-848b5650dfec-podres\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.696302 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.696267 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6254fb31-ae08-438e-93cc-848b5650dfec-lib-modules\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.696302 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.696298 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwq79\" (UniqueName: \"kubernetes.io/projected/6254fb31-ae08-438e-93cc-848b5650dfec-kube-api-access-fwq79\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.696553 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.696322 5532 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6254fb31-ae08-438e-93cc-848b5650dfec-sys\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.696553 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.696358 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/6254fb31-ae08-438e-93cc-848b5650dfec-podres\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.696553 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.696403 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6254fb31-ae08-438e-93cc-848b5650dfec-sys\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.709482 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.709455 5532 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwq79\" (UniqueName: \"kubernetes.io/projected/6254fb31-ae08-438e-93cc-848b5650dfec-kube-api-access-fwq79\") pod \"perf-node-gather-daemonset-pp75c\" (UID: \"6254fb31-ae08-438e-93cc-848b5650dfec\") " pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.718665 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.718641 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5fj4_896e7579-dea0-49ee-92c5-cddf258c6905/node-exporter/0.log" Oct 31 14:22:51.737371 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.737346 5532 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:51.743566 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.743545 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5fj4_896e7579-dea0-49ee-92c5-cddf258c6905/kube-rbac-proxy/0.log" Oct 31 14:22:51.768773 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.768745 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p5fj4_896e7579-dea0-49ee-92c5-cddf258c6905/init-textfile/0.log" Oct 31 14:22:51.889697 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.889583 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-949gv_3e3f5e50-ca1a-4a2a-af47-30a240b36bd5/kube-rbac-proxy-main/0.log" Oct 31 14:22:51.915743 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.915714 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-949gv_3e3f5e50-ca1a-4a2a-af47-30a240b36bd5/kube-rbac-proxy-self/0.log" Oct 31 14:22:51.941937 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.941911 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-949gv_3e3f5e50-ca1a-4a2a-af47-30a240b36bd5/openshift-state-metrics/0.log" Oct 31 14:22:51.983105 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:51.983078 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e683f34-6fee-4e68-a479-6688ab8221c3/prometheus/0.log" Oct 31 14:22:52.008790 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.008763 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e683f34-6fee-4e68-a479-6688ab8221c3/config-reloader/0.log" Oct 31 14:22:52.016950 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.016928 5532 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c"] Oct 31 14:22:52.019387 ip-10-0-128-41 kubenswrapper[5532]: W1031 14:22:52.019361 5532 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6254fb31_ae08_438e_93cc_848b5650dfec.slice/crio-73e19cbbe96b652ffd971f6a79ac490b4d48893e5bdfa929c6cb19ecd99d076c WatchSource:0}: Error finding container 73e19cbbe96b652ffd971f6a79ac490b4d48893e5bdfa929c6cb19ecd99d076c: Status 404 returned error can't find the container with id 73e19cbbe96b652ffd971f6a79ac490b4d48893e5bdfa929c6cb19ecd99d076c Oct 31 14:22:52.038150 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.038128 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e683f34-6fee-4e68-a479-6688ab8221c3/thanos-sidecar/0.log" Oct 31 14:22:52.068053 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.068031 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e683f34-6fee-4e68-a479-6688ab8221c3/kube-rbac-proxy-web/0.log" Oct 31 14:22:52.098164 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.098142 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e683f34-6fee-4e68-a479-6688ab8221c3/kube-rbac-proxy/0.log" Oct 31 14:22:52.125644 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.125619 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e683f34-6fee-4e68-a479-6688ab8221c3/kube-rbac-proxy-thanos/0.log" Oct 31 14:22:52.165323 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.165303 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_0e683f34-6fee-4e68-a479-6688ab8221c3/init-config-reloader/0.log" Oct 31 14:22:52.266009 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.265975 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-5vhhj_25233e95-abc9-4f04-b7b2-65933229d471/prometheus-operator-admission-webhook/0.log" Oct 31 14:22:52.302175 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.302147 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-96946754f-9jtzl_ddd6aed9-e9c4-4403-a941-bcb9020faac0/telemeter-client/0.log" Oct 31 14:22:52.330737 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.330716 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-96946754f-9jtzl_ddd6aed9-e9c4-4403-a941-bcb9020faac0/reload/0.log" Oct 31 14:22:52.357377 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.357357 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-96946754f-9jtzl_ddd6aed9-e9c4-4403-a941-bcb9020faac0/kube-rbac-proxy/0.log" Oct 31 14:22:52.392558 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.392514 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c987d54b-lc452_e25856e6-e50d-48d2-94a5-8966be78dca4/thanos-query/0.log" Oct 31 14:22:52.393389 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.393367 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" event={"ID":"6254fb31-ae08-438e-93cc-848b5650dfec","Type":"ContainerStarted","Data":"b0e2345782db81d3969c6487ff98dc7e30b648994d2545fd3452548875de26f7"} Oct 31 14:22:52.393507 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.393416 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:22:52.393507 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.393438 5532 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" event={"ID":"6254fb31-ae08-438e-93cc-848b5650dfec","Type":"ContainerStarted","Data":"73e19cbbe96b652ffd971f6a79ac490b4d48893e5bdfa929c6cb19ecd99d076c"} Oct 31 14:22:52.414806 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.414730 5532 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" podStartSLOduration=1.4147173020000001 podStartE2EDuration="1.414717302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 14:22:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 14:22:52.413420538 +0000 UTC m=+4740.559164367" watchObservedRunningTime="2026-10-31 14:22:52.414717302 +0000 UTC m=+4740.560461130" Oct 31 14:22:52.422579 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.422562 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c987d54b-lc452_e25856e6-e50d-48d2-94a5-8966be78dca4/kube-rbac-proxy-web/0.log" Oct 31 14:22:52.450753 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.450728 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c987d54b-lc452_e25856e6-e50d-48d2-94a5-8966be78dca4/kube-rbac-proxy/0.log" Oct 31 14:22:52.527035 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.527006 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c987d54b-lc452_e25856e6-e50d-48d2-94a5-8966be78dca4/prom-label-proxy/0.log" Oct 31 14:22:52.573989 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.573963 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c987d54b-lc452_e25856e6-e50d-48d2-94a5-8966be78dca4/kube-rbac-proxy-rules/0.log" Oct 31 14:22:52.606713 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:52.606685 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57c987d54b-lc452_e25856e6-e50d-48d2-94a5-8966be78dca4/kube-rbac-proxy-metrics/0.log" Oct 31 14:22:54.940777 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:54.940748 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6bb896f648-h8cht_80cc2424-4b62-4667-b63d-ad7bca01c413/console/0.log" Oct 31 14:22:54.997569 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:54.997545 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-xc5m4_70ea50fe-3024-4a82-84a5-5c236706402b/download-server/0.log" Oct 31 14:22:56.843353 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:56.843306 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7rzwh_ea0c07d8-f966-4acb-aea6-1ec8682b4877/dns/0.log" Oct 31 14:22:56.867145 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:56.867121 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7rzwh_ea0c07d8-f966-4acb-aea6-1ec8682b4877/kube-rbac-proxy/0.log" Oct 31 14:22:57.047330 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:57.047298 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l5vvf_dc1ff283-237c-45f3-a90f-88cb75bcb863/dns-node-resolver/0.log" Oct 31 14:22:57.763910 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:57.763870 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fbstt_54a25982-6507-405a-8e10-81d965ab7f4f/node-ca/0.log" Oct 31 14:22:58.409512 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:22:58.409478 5532 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bkxgk/perf-node-gather-daemonset-pp75c" Oct 31 14:23:01.761440 ip-10-0-128-41 kubenswrapper[5532]: E1031 14:23:01.761359 5532 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-nlmx2" containerName="extractor" Oct 31 14:23:02.469084 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:02.469058 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7485898565-kp66f_c87eda91-1ec7-42bf-bf88-250b804bd332/kube-auth-proxy/0.log" Oct 31 14:23:02.620230 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:02.620179 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-76bb5_7d461f31-7477-4c4f-81fa-6a4769e2266f/istio-proxy/0.log" Oct 31 14:23:02.727818 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:02.727719 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-hdtr7_b13b4423-6b52-4dd7-be75-987fd6e2edfa/istio-proxy/0.log" Oct 31 14:23:02.797833 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:02.797800 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-qhgv4_fdf3bdf7-1bcc-43e5-bccd-a375cce224fd/istio-proxy/0.log" Oct 31 14:23:02.833464 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:02.833404 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-tvm4p_532959bd-96e4-40f7-a157-7302d6216a41/istio-proxy/0.log" Oct 31 14:23:05.601883 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:05.601848 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hgm5n_e507f475-28b0-43c1-86b1-5ee3e4c4dad4/serve-healthcheck-canary/0.log" Oct 31 14:23:06.008417 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:06.008379 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7485898565-kp66f_c87eda91-1ec7-42bf-bf88-250b804bd332/kube-auth-proxy/0.log" Oct 31 14:23:06.095935 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:06.095866 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-76bb5_7d461f31-7477-4c4f-81fa-6a4769e2266f/istio-proxy/0.log" Oct 31 14:23:06.154696 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:06.154666 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-hdtr7_b13b4423-6b52-4dd7-be75-987fd6e2edfa/istio-proxy/0.log" Oct 31 14:23:06.188441 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:06.188409 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-qhgv4_fdf3bdf7-1bcc-43e5-bccd-a375cce224fd/istio-proxy/0.log" Oct 31 14:23:06.206315 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:06.206291 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5d84bc6b95-tvm4p_532959bd-96e4-40f7-a157-7302d6216a41/istio-proxy/0.log" Oct 31 14:23:06.363460 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:06.363387 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nlmx2_a0b84e1d-390d-434d-b309-9440ec8195b6/kube-rbac-proxy/0.log" Oct 31 14:23:06.392476 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:06.392448 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nlmx2_a0b84e1d-390d-434d-b309-9440ec8195b6/exporter/0.log" Oct 31 14:23:06.421079 ip-10-0-128-41 kubenswrapper[5532]: I1031 14:23:06.421051 5532 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-nlmx2_a0b84e1d-390d-434d-b309-9440ec8195b6/extractor/0.log"